NetFlow Ninjas

University Internet Usage on the Rise: Is Your ResNet Ready?

Nielsen's Law of Internet Bandwidth states that a high end user's Internet connection speed will grow 50% per year. Few organizations experience the implications of this law more than higher education, especially at the beginning of a new school year.

To get a feel for just how much Internet an average university consumes we asked a few of our EDU customers to grab screenshots of their Internet utilization resulting from the return of students to the campus after 2010 summer break. The results are as expected, massive uptick in overall Internet as students return to the campus ResNet. "ResNet" is the term EDUs give the "Residential Network" in which students live, study, and play. The charts below were generated using Lancope’s StealthWatch NetFlow/sFlow Collection System and show deduplicated traffic stats broken down by layer-4 port number.

 
#1 This screenshot shows a 400% increase in overall bandwidth from one week to the next...

 

#2 Here's a busy EDU environment with bandwdith approaching 3G. Once all the students are back utilization will likely hit 4Gbps. Note the drop-off of bandwidth during the weekends as students and faculty leave the campus...

 

#3 The brown area at the bottom shows lots of UDP traffic associated with chatty peer-to-peer network traffic. Again note the upward climb as students return to the campus...

#4 Everything runs over HTTP these days, this EDU chart below demonstrates this fact quite clearly. Interestingly this EDU saw very little drop over the weekend prior to the new semester starting. 

The overall upward trend in Internet use is the result of a few generic trends in Internet technology:

1. HD video content – HD video consumes a ton of bandwidth, 2Mbps+ for some streams. Many consumer devices actually capture video in 720p or better and the trend is toward even higher resolutions in hand-held devices. The result is lots of high-res, high bandwidth content arriving at ResNet. Check out this blog for an example.
2. Smartphones – In addition to the student’s laptop or desktop smartphones such as the HTC Incredible and the iPhone are literally blowing the Internet up. Ask AT&T users in San Fran’s Bay Area. Most smartphones can be connected to the local WLAN, bypassing 3G connections and using the university’s network instead.
3. Peer-to-peer file sharing – Legal aspects aside, students love BitTorrents, Kazaa, and other peer-to-peer content distribution technologies. These  protocols come at a heavy price to the campus backbone in both bits per second and concurrent connections. 
4. Social Media – To a lesser degree but still relevant is the Internet traffic created by social media applications such as Facebook. Facebook was literally created for EDUs so it's no surprise it carries such popularity in student social circles.

As a network university network administrator, you should be using flow technology to gain a better understanding of your network utilization growth over time. NetFlow provides the information you need to plan and budget for the arrival of students in the new year.

 

QUICK ASIDE: As a long time Mac user, I find it interesting that three of the four screenshots above (the 1st, 2nd, and 3rd) are from a Mac. University admins and students love the Mac. It would be nice to see more Mac use in the enterprise space. Too bad Apple fails at marketing to the enterprise buyer. In my experience Macs have less issues than their PC counterparts which is great for students and their network administrators given the wild-west nature of the ResNet environment. Not to show off or anything but how many laptop PC users can claim uptime values of 29 days or more? Just sayin...

 

Can You Secure Cisco's Borderless Network?

Here’s an interesting article from SearchNetworking’s Shamus McGillicuddy on Cisco’s enigmatic network security strategy.  Clearly end users at organizations are concerned with the security implications of Cisco’s Borderless Network campaign, but the reality is that this is just an evolution of network security in general.  As borderless networks become the norm with the introduction of IT consumerization, the trend toward mobility, applications moving to the cloud, and an increased demand for video and telepresence support, organizations must find a means to monitor the increasingly complex and amorphous network.

 

Cisco’s Borderless Network initiative anticipates these changes and forces a spirit of collaboration not only among network and security managers but also among network and security vendors.  Conventional operational silos and point solutions are becoming irrelevant.  Former rivals (whether internal users or vendors) can now become partners as various users within the IT organization work together to solve all performance issues – whether network, security or application related.

 

Regardless of previous initiatives, Cisco is now streamlining its focus on core security technologies to provide a comprehensive strategy that maximizes existing technology investments.  Case in point, Cisco IOS NetFlow has become an increasingly popular option for organizations to gain borderless network visibility  – providing pervasive coverage with minimal acquisition costs.  Organizations with concerns about Cisco’s future in network security should consider leveraging their existing Cisco infrastructure to improve network performance and security management.  Besides, no one ever got fired for buying Cisco.

  

NetFlow Events Rates vs. Actual Network Traffic: Not Always What You Expect

I've blogged about this topic before but wanted to circle around for a quick update on the topic with an interesting anecdote from a large university with high bps, pps, and fps rates. Take a look at the charts below. Each are from the exact same time period but show the university's traffic in different ways. The top chart shows the last 20 hours of traffic in overall bits per second, the second chart shows the same traffic in packets per second, and the final chart (in red) shows the number of NetFlow flows per second generated by the same network traffic...

The top two charts show the normal network utilization curve associated with human circadian rhythms (times are shown in GMT so subtract 5 hours). Traffic levels are at their lowest around 2:00 to 4:00 am.

Note the area of the chart marked in yellow. The overall shape of the Flows Per Second chart looks almost nothing like the above two charts. This actually caught me somewhat off guard as I expected the flow rate to match the packet rate quite closely. It turns out that the NetFlow export point at this particular university is outside their Intrusion Prevention System and router-based access lists. The charts shows that the university is seeing 25,000 flows per second even during off peak hours (3am). The charts include all the random, high connection rate traffic from scanning, botnets, bittorrents, DoS attacks and other automated Internet applications (aka "Dirty Flows")...

These automated applications don't sleep but also don't create all that much actual network volume compared to a real human accessing Facebook, Youtube, iTunes, etc. The result is a sustained flow rate throughout the day but a dip in overall bps and pps when humans aren't interacting with the network.

Somewhat difficult concept to grasp at first but makes sense when you think about it for a bit. The key take away here is that NetFlow event rates don't necessarily correlate with bps or pps network levels. Especially when you're dealing with NetFlow exporters that are outside the perimeter firewall. 

So when your sizing up your NetFlow collector take into consideration the nature of the network your monitoring. Check out this post for more info on key considerations.

          

Ultrinsic.com: Is Sharing Your Student Login Really a Good Idea?

Lancope does a *lot* of business with Higher Education so when a colleague brought this new service to my attention I just had to mention it here.

From Ultrinsic.com's website:

"Ultrinsic is a web-based college platform that provides incentives to students for academic achievement. Ultrinsic exclusively dedicates itself to motivating students to improve their grades."

So how do they "motivate students to improve their grades"? By betting on success or failure of course! The idea itself is quite interesting...

1. The student signs in and places bets or "Contributions" on their ability to achieve a minimum grade for a specific class.                  (shown at right) ------->

2. Ultrinsic will match the Contribution based on the student's previous academic history.

3. If the student makes the grade they get paid, if they miss their Target Grade they lose their Contribution.

The irony here is that Ultrinsic actually wants the student to miss their expectations since their model requires you miss the Target Grade in order for them to make money. 

Anyway, I'm not really all that bothered by the business model itself since there are far worse activities on which students can waste their parent's money. The real issue here is the method Ultrinsic uses to verify the grades...

From Ultrinsic's FAQ: 

How do I verify my grades?
You can verify your grades by sending an official transcript directly from the Registrar's Office or provide access to your online school account.

Yikes! Ultrinsic wants you to hand over your student enrollment services login. This sounds like all kinds of bad. We know almost nothing about this fledgling company and the product isn't even out of beta yet but they want you to surrender a login that potentially contains gobs of personal information.

     

I was curious what sort of information is available from a typical student enrollment system login these days so I borrowed my brother's Vanderbilt login and here's the first thing I see...

Now maybe I'm overreacting here but the problem seems to be that in order to enable automated verification of grades we also have to grant access to "Student Accounts" and "Financial Aid". Vanderbilt's single-signon student portal isn't on Ultrinsic's list of supported schools but a bunch of major schools are including...

University of California-Berkeley
University of Southern California
Stanford University
University of California-Los Angeles
University of Connecticut
Howard University
American University
George Washington University
Georgetown University
Indiana University-Bloomington
Brandeis University
Boston College
Harvard University
Boston University
Michigan State University
University of Michigan-Ann Arbor
North Carolina State University
University of North Carolina
Wake Forest University
Duke University
Rutgers University
CUNY Queens College
SUNY Binghamton
Syracuse University
St. Johns University
New York University
Columbia University
Pennsylvania State University
University of Pittsburgh
University of Pennsylvania
Princeton University
University of Texas-Austin
Texas Tech University
Texas A&M University
Brigham Young University
University of Wisconsin-Madison

So what does all this have to do with NetFlow? Well the great thing about Lancope's StealthWatch System is that every single flow created on the network is recorded and stored for later retrieval and analysis.

If you're a university administrator and want to quickly see how many of your students are accessing this service you can just run a Flow Summary Report against Ultrinsic's web-hosting subnet range of 199.103.56.0/24. You'll see something like this...

My opinion: It's never a good idea to give out login info to any service that intends to scrape data for automated purposes such as these. Maybe if it's an extremely well trusted source but some random startup who's not even out of beta? I'll pass.

     

New Book: Network Flow Analysis

Check out Chris Durkin's "The Unofficial MARS Blog" and his book review of Network Flow Analysis by Michael W.Lucas, in which he adds a nice shout out for StealthWatch by Lancope. 

Thanks, Mark Gatehouse - a fellow NetFlow Ninja - for alerting us to this interesting blog post.

Lancope and Damballa Host Free Webinar on How to Use NetFlow to Battle Advanced Cyber Threats

Lancope, Inc.,  the leader in NetFlow™ collection and analysis and the provider of the StealthWatch® System for flow-based network performance and security monitoring, and Damballa, the company transforming the fight against cyber threats, today announced that they will present a live Webinar entitled “Use NetFlow to Detect and Stop Advanced Cyber Threats.” Damballa Vice President of Product Management Steven Newman and Lancope Product Manager Joe Yeager will lead the Webinar on Tuesday, August 17, at 11 a.m. ET.

Register for this free event at: http://www2.lancope.com/webinar_registration_august.

Get more details on this webcast at: http://news.lancope.com/bid/28763/Lancope-and-Damballa-Host-Free-Webinar-on-How-to-Use-NetFlow-to-Battle-Advanced-Cyber-Threats.

More Movement in the NetFlow Arena: Arbor Networks Acquired by Tektronix

Congrats to my friend Roland Dobbins and the other folks over at Arbor Networks for their announced acquisition by Tektronix Communications. The deal amount was undisclosed but I've got to imagine they came at a pretty penny given Arbor's continued success in the service provider and cloud markets.

I always had a lot of respect for the Arbor guys and considered them to be the "ISP NetFlow Ninjas". Where Lancope has always focused on NetFlow in the enterprise, Arbor focused sharply on NetFlow in the large service provider market. We rarely meet in the field.

For the sake of SPs everywhere that rely on Arbor's products let's hope the Tektronix purchase doesn't take their eye off the ball as it has some other vendors in the NetFlow space *cough* Mazu *cough*.

        

Netflow-based StealthWatch by Lancope Named Best Security and Network Management Solution in Enterprise Management Associates Application-Aware Network Performance Management Radar Report

 Lancope, Inc, the leader in NetFlow™ collection and analysis and flow-based network performance and security monitoring, today announced that independent analyst firm Enterprise Management Associates (EMA) recognizes its flagship technology – the StealthWatch® System – as the IT industry’s best joint security and network management solution. EMA’s Radar™ for Application-Aware Network Performance Management Q3 2010 report also names StealthWatch a strong value for its product strength and cost efficiency. Read more

 
 
  Download Report  [registration required]

Upsetting the Old School with NetFlow Innovation in Application Performance Monitoring

EMA's new report on Application-Aware Network Performance Management (ANPM) highlights a few newcomers to the space. CA, NetScout, OPNET, and the other members of the network monitoring old school should probably watch their backs because NetFlow-centric vendors are making major headway.    

"Lancope is a stable and growing private company with a firm vision of the role that their solutions play in response to ANPM requirements, and a core strength in security monitoring which clearly differentiates them from other ANPM vendors. They also have an aggressive roadmap for filling functionality gaps and solid partnerships with complementary network equipment and management providers which will pay dividends in terms of extended solution value over the long term. Consequently, EMA considers Lancope to be a highly stable and reliable ANPM solution provider."

Jim Frey at EMA does a great job drilling down on each of the major players in the network performance monitoring space and lays out the current landscape quite nicely in their Radar Report shown below...

The important thing to note here is that Lancope wasn't even on this chart 18 months ago but now we're sitting high in the "Strong Value" category. With the release of our 6.0 product early next year (more to come on 6.0 soon!) Lancope will push directly into the "Value Leader" realm.

Our strategy has been and remains: Maintain our leadership in security analysis of flow data while continuously pushing into the performance monitoring side of network operations. We're bringing NetFlow collection, network behavioral analysis, and affordable continuous monitoring FlowSensor probe technology together into a single platform that provides unrivaled visibility into how your network is being used and where the potential threats lie.

NetFlow technology is key to the next generation of network performance monitoring. New NetFlow features like NBAR allow NetFlow-oriented vendors such as Lancope to monitor application performance without the grossly expensive probes required by other vendors in the space. If you don't have NetFlow but need deep network monitoring we offer the FlowSensor network appliance. The FlowSensor creates NetFlow in areas of the network where you don't already have support. It also calculates latency and allows for ad-hoc packet captures. All while using industry standard NetFlow v9.

Here's the full report. Registration is required but it only takes a few seconds so check it out...

Download EMA's Report on Application Aware network Performance Monitoring

       

Cost Comparison: NetFlow vs. Packet Capture

A lot of people ask us:

"Cost wise, how does NetFlow compare to traditional packet capture technologies that leverage SPAN ports or Ethernet taps?"

Well let's run though a quick US list price comparison of Lancope's StealthWatch enterprise product line vs. NetScout's InfiniStream network probe solution. Let's assume the customer wants to monitor 100 locations each at 1Gbps...

Lancope StealthWatch Management Console = $29,995
Lancope StealthWatch Xe-1000 NetFlow Collector = $109,995 (w/100 sources)

NetScout nGenius Performance Manager Global Server = $57,659.99
NetScout nGenius InfiniStream 2906/GS = $2,094,199 (100 units @ $20,941.99 each)

Lancope StealthWatch w/100 NetFlow sources:   $139,990
NetScout's nGenius w/100 gigabit probes:       $2,151,858

That's a 15:1 cost ratio of NetFlow over Packet Capture. Compelling? Yes indeed.

Sorry NetScout, I hate to pick on you like this but your pricing is listed in plain sight on CDW and to be honest your an easy target.

And before anyone calls me on it, yes I understand that there are major feature differences between NetFlow and packet capturing systems. Native NetFlow doesn't provide latency statistics, session reassembly, or full packet decodes but it does provide the data needed to construct top talker lists of all types, elaborate traffic breakdowns, router interface utilization statistics, trending information, and advanced anomaly detection and alerting capabilities.

The point here is that you don't need heavy-weight packet capture technology everywhere throughout your network if you have a capable NetFlow collection system. You probably couldn't afford it even if you did need it. Keep your expensive "packet library" technology at key locations such as Internet gateways and use NetFlow everywhere else where you need network visibility.

BTW: If you really must have latency and packet capture capabilities, Lancope offers a device called a FlowSensor which plugs into a SPAN, tap, or mirror port to generate NetFlow suitable for consumption by any NetFlow v9 capable collector. The FlowSensor is priced at about 20% the cost of other probe solutions.

      

Quick Off Topic: I was on vacation all last week playing Jacque Cousteau at my dad's ranch in central Florida. Beautiful countryside and even more amazing sea life. Check out this funny video I made catching sea scallops off the Florida gulf coast at Crystal River. Also this other beautiful video  I put together shows off one of Florida's natural springs called the Rainbow River. In case your wondering I didn't see *any* oil out in the gulf. If it's there it's hiding.

           

Facebook’s 500 Millionth Member Highlights Risks to Corporations

Facebook’s recent 500 million account milestone illustrates the explosive growth of social media technology within the Internet community. The current trend points toward 1 billion accounts within 12 months. 

Organizations must consider what Facebook, Linkedin, Twitter, blogs, and other social media mean to the organization, form a Social Media Policy, and educate their users. Access to social media networks such as Facebook and Linkedin have proven benefits such as improved moral, collaboration, and administrative efficiency. Unfortunately in some organizations the risks could outweigh the value.

IT administrators should be aware of the following productivity and security issues associated with social networking sites such as Facebook: 

       

1. Workplace Productivity Issues – Facebook Chat and games such as Mafia Wars and Farmville can add up to huge losses in worker productivity if abused. Corporations must adopt a policy on the use of social networking applications, tracking user access and blocking certain sites deemed high-risk. Once your Social Media Policy is established, make sure to educate your users on what the policy means to their daily lives.

The list below shows overall Facebook usage by Lancope employees over a 24 hour period. It's pretty easy to spot excessive Facebook usage. MafiaWars and other high traffic games will run the byte count up quick. Check out how far above me (Adam) the top user is... Excessive Facebook usage? Probably.

           
Also, what is your policy on instant messaging? Programs such as Trillian and Adium can connect directly to Facebook allowing users to chat with other logged-in Facebook users without even opening a browser. Facebook Chat uses HTTP over TCP port 80 so it looks like any other web traffic. Fortunately you cannot perform file transfers over Facebook Chat at this time.

 

   

2. Phishing and Social Engineering - Attackers can leverage personal information found on social networking sites to coax victims into giving away confidential or proprietary corporate information. Again, education is the best defense against social engineering attacks through social media.

Twitter posts can be up 140 characters in length. This forces users to use URL shortening services such as bit.ly, tinyurl.com, and is.gd. Shortened URLs hide the original domain name and link values from the user, increasing the likelihood that a user will be tricked into access a site that contains malicious code or spoofs an otherwise legitimate site.

Users need to well educated on the possibility that they could be scammed by fake Facebook profiles seeking to coax you into disclosing confidential information. A recent experiment by consultant Thomas Ryan showed that even seasoned cyber security professionals are often no match for "hot chicks". Ryan's fictions femme fatale got dinner invites, job offers, and speaking opportunities at several trade shows out of workers at such organizations as the National Reconnaissance Office and the DoD's Joint Chiefs of Staff. If these professionals can be fooled, you can bet social media users not into cyber security technology would be fooled even easier.

Full story at the Washington Times here.

    

     



3. Flash-based Vulnerabilities in Games – Web 2.0 technology has led to an amazingly rich and unfortunately fragile world of games and web-based apps – many of which run in Facebook’s social media platform. Attack vectors in enabling technology such as Flash have led to multiple incidents over the last few years. Regardless of whether workers are using Flash-based games in the office or at home, if they use a wireless device to access a corporate network and social networking applications, they are putting their company at risk. A policy against flash-based gaming and a rigid patch management process pay off well here.



If you use Lancope's StealthWatch NetFlow collection system you can set triggers against Farmville usage then get alerts when users access Farmville servers.   

      

4. Information Leakage – Corporations need to reinforce their policies on proprietary information disclosure into the social media meta-space. Ensuring private company information or SEC regulated information doesn’t make its way into the public’s hands is a must.

 Social media information disclosure is more a hypothetical at this point as the SEC hasn't officially investigated such an incident to date. I would hate my company to be the first. 

    

5. Network Impact – Organizations with large concentrations of users can see a significant network impact from excessive use of social media, especially YouTube and Facebook. The clicky nature of Facebook games leads to many concurrent connections active from each user which leads to memory and bandwidth consumption across the network. A balance needs to be struck between workers using social networking applications for work and/or for personal use. Firewall, proxy log, and NetFlow logs are excellent tools for measuring the impact from social media usage.

The chart below shows the aggregate Facebook traffic generated by about 30 Facebook users at Lancope over the last 7 days...

           

So the important take-aways here are:

1. Think the various risks social media presents to your business.

2. Create a policy the negates as much risk as is practical without running off your employees.

3. Educate your users on the policy.

4. Verify with technology where possible (NetFlow!)