Scaling Synchronous Web Apps: Lessons learned from Meebo
Sandy Jen, co-founder of Meebo, is on the stage talking about Scaling Synchronous Web Apps at Web 2.0 Expo. There are about 40 million unique people using it and they are around 50 people or so.
The presentation is about tips and practices they have learned about how to scale now vs in 3 years. Scalability is very subjective.
the "hole"
- multi-platform (lots of browsers)
- spotty network connections
- there are only 2 simultaneous open HTTP requests allowed (for now)
- page views: Meebo uses 1 page so it is really hard to measure unless you do it on your own. On average a person stays on Meebo for two hours.
- static content
- no downloads
the "peg": the thing that you're trying to shove into the hole.
She is more involved with the server side so the talk will be server focused.
Don't underestimate the server side architecture! In the start you won't know where the bottlenecks will be.
The type of application you are building determines the type of synchronous scaling you are going to need.
Peg helpers:
Can you use FastCGI vs. web modules vs. PHP? They ended up writing a module directly into the server.
Do you need to save state? is it persistent? Meebo didn't have user accounts for a long time. Launching feature light is not a bad thing.
Tug of war: There is always a delicate balance between what you can push to the browser and what you can let the server handle. Ask yourself where does the workload make sense? Browsers can be SLOW (just because you can do it in JavaScript, doesn't mean you should). Meebo users mostly use IE so it's optimized for that. Your users will sadly know more about your product than you do because they use it more. Also don't ignore the importance of efficiency with data transfer.
Good enough vs perfect: Perfection is enough simplicity in the system to allow for adaptation. Release and iterate often.
Think ahead but not too much: You shouldn't over-design for the unknown as over-designed code can be worse than hacky code. It makes it difficult to roll back an entire design. Think about building horizontally and see if you can throw more servers at the problem initially? Adding servers is expensive but has a shorter lead time, architecture takes longer but is "free" (development time is not free). Think about striking the right balance. You won't know where your bottlenecks are until you let it use.
Nothing simulates real life: You're probably not the end user. Contingency plans are key. Have some tricks that you haven't used yet. You should have a few tricks up your sleeve. In the end you'll always miss something, but that's ok.
Don't build flood gates, but instead build dams so you will be able to enable/disable components or change operating parameters during runtime. This includes front-end and back-end. When you roll out users, let them know that it may be buggy and you may need to pull it back.
In the end your users will behave in ways you never imagined.
Use your own product: don't be afraid to find bugs. You should overcome the fear of breaking your own product. It helps to keep your finger on the pulse of the community.Trust your users. Use blogs and scale customer service appropriately to stay in touch with users. 70% of meebo's users use IE. Make sacrifice for your users and use what they are using. Go where your users are going.
Being Big Brother: You should be aware of what is going on and monitor key areas: zabbix, nagios, hyperic, ganglia, etc. Eventually you WILL have to build something custom. Don't go overboard on monitoring, you're going to learn to ignore your alerts. You shouldn't ignore what your system tell you (not just hardware) as it is extremely dangerous. Goal of monitoring is basically asking: "is your app healthy?"
Final thoughts: There's no magic solution to scalability. It's very important that you understand the nooks and crannies of your architecture. You should be able to correlate effects to any system changes you make. Always keep the goal in mind, don't lose sight of it. What are you scaling? Remember, everyone scales differently!
The presentation is about tips and practices they have learned about how to scale now vs in 3 years. Scalability is very subjective.
the "hole"
- multi-platform (lots of browsers)
- spotty network connections
- there are only 2 simultaneous open HTTP requests allowed (for now)
- page views: Meebo uses 1 page so it is really hard to measure unless you do it on your own. On average a person stays on Meebo for two hours.
- static content
- no downloads
the "peg": the thing that you're trying to shove into the hole.
- instantaneous data transfer: How do you deal with that?
- long polling: browsers suck up memory
- making the browser do work: Javascript requires the browser to do something. Trying to balance what you make the browser do and what you let the server do is really important.
- seamless user experience: you're not going to use a product you don't like. Overall that's the thing you want to strive for.
She is more involved with the server side so the talk will be server focused.
Don't underestimate the server side architecture! In the start you won't know where the bottlenecks will be.
The type of application you are building determines the type of synchronous scaling you are going to need.
Peg helpers:
- long polling (COMET): to keep connections open with Web servers
- web servers: using the right technology. Meebo started with Apache which turned out to be not very good. They tried lighttpd and it turned out to be a great help as lighttpd is single thread and event based
- compiled vs. interpreted: Meebo is written in C/C++. It's a tradeoff. hiring is difficult for them
- databases: can be really expensive or cheap. They use MySQL. Their schemas are simple.
- memcache:
- load balancers: really expensive and you have to buy them in pairs.
Can you use FastCGI vs. web modules vs. PHP? They ended up writing a module directly into the server.
Do you need to save state? is it persistent? Meebo didn't have user accounts for a long time. Launching feature light is not a bad thing.
Tug of war: There is always a delicate balance between what you can push to the browser and what you can let the server handle. Ask yourself where does the workload make sense? Browsers can be SLOW (just because you can do it in JavaScript, doesn't mean you should). Meebo users mostly use IE so it's optimized for that. Your users will sadly know more about your product than you do because they use it more. Also don't ignore the importance of efficiency with data transfer.
Good enough vs perfect: Perfection is enough simplicity in the system to allow for adaptation. Release and iterate often.
Think ahead but not too much: You shouldn't over-design for the unknown as over-designed code can be worse than hacky code. It makes it difficult to roll back an entire design. Think about building horizontally and see if you can throw more servers at the problem initially? Adding servers is expensive but has a shorter lead time, architecture takes longer but is "free" (development time is not free). Think about striking the right balance. You won't know where your bottlenecks are until you let it use.
Nothing simulates real life: You're probably not the end user. Contingency plans are key. Have some tricks that you haven't used yet. You should have a few tricks up your sleeve. In the end you'll always miss something, but that's ok.
Don't build flood gates, but instead build dams so you will be able to enable/disable components or change operating parameters during runtime. This includes front-end and back-end. When you roll out users, let them know that it may be buggy and you may need to pull it back.
In the end your users will behave in ways you never imagined.
Use your own product: don't be afraid to find bugs. You should overcome the fear of breaking your own product. It helps to keep your finger on the pulse of the community.Trust your users. Use blogs and scale customer service appropriately to stay in touch with users. 70% of meebo's users use IE. Make sacrifice for your users and use what they are using. Go where your users are going.
Being Big Brother: You should be aware of what is going on and monitor key areas: zabbix, nagios, hyperic, ganglia, etc. Eventually you WILL have to build something custom. Don't go overboard on monitoring, you're going to learn to ignore your alerts. You shouldn't ignore what your system tell you (not just hardware) as it is extremely dangerous. Goal of monitoring is basically asking: "is your app healthy?"
Final thoughts: There's no magic solution to scalability. It's very important that you understand the nooks and crannies of your architecture. You should be able to correlate effects to any system changes you make. Always keep the goal in mind, don't lose sight of it. What are you scaling? Remember, everyone scales differently!
Labels: meebo, sandy-jen, scalability, synchronous, web 2.0, web2expo
posted by Frank at 1:47 PM
0 comments
