2010-09-03 Fri

07:46 Hot Scalability Links For Sep 3, 2010 (1867 Bytes) » High Scalability

With summer almost gone, it's time to fall into some good links...

Hibari - distributed, fault tolerant, highly available key-value store written in Erlang. In this video Scott Lystig Fritchie gives a very good overview of the newest key-value store.
Tweets of Gold
- lenidot: with 12 staff, @tumblr serves 1.5billion pageviews/month and 25,000 signups/day. Now that's scalability!
- jmtan24: Funny that whenever a high scalability article comes out, it always mention the shared nothing approach
- mfeathers: When life gives you lemons, you can have decades-long conquest to convert lemons to oranges, or you can make lemonade.
- OyvindIsene: Met an old man with mustache today, he had no opinion on #noSQL. Note to myself: Don't grow a mustache, now or later.
- vlad003: Isn't it interesting how P2P distributes data while Cloud Computing centralizes it? And they're both said to be the future.
You may be interested in a new DevOps Meetup organized by Dave Nielson, so you know it will be good.

06:15 Check-in, Chicken (1354 Bytes) » Seth's Blog

One way to start every morning with your team is to have them check in. Go around in a circle and let people update and contribute. It's not a silly exercise, in that it helps people speak up and it communicates forward motion.

Another way, probably a better one, is to have each member of the team announce what they're afraid of. Two kinds of afraid, actually. Things that might fail and things that might work.

What are you, chicken?

Yes, we're chicken. We're afraid. The lizard has us by the claws.

So, tell us. What are you afraid might happen that would destroy, disintegrate, or dissuade--that would take us down? And what are you afraid of that might work, thus changing everything and opening up entirely new areas of scariness?

02:37 Six guiding principles to Consolidate your IT (647 Bytes) » High Scalability

The need for IT consolidation is most evident in two types of organizations. In the first group, IT grew organically with business over the decades, and survived changes of strategy, management, staff and vendor orientation. The second group of businesses capital groups are characterized by rapid growth through acquisitions (followed by attempts to integrate radically different IT environments). In both groups, their IT infrastructures have typically been pieced together over the past 20 (or more) years.

2010-09-02 Thu

14:40 How long Innodb Shutdown may take (3730 Bytes) » MySQL Performance Blog

How long it may take MySQL with Innodb tables to shut down ? It can be quite a while.
In default configuration innodb_fast_shutdown=ON the main job Innodb has to do to complete shutdown is flushing dirty buffers. The number of dirty buffers in the buffer pool varies depending on innodb_max_dirty_pages_pct as well as workload and innodb_log_buffer_size and can be anywhere from 10 to 90% in the real life workloads. Innodb_buffer_pool_pages_dirty status will show you the actual data. Now the flush speed also depends on number of factors. First it is your storage configuration – you may be looking at less than 200 writes/sec for single entry level hard drive to tens of thousands of writes/sec for high end SSD card. Flushing can be done using multiple threads (in XtraDB and Innodb Plugin at least) so it scales well with multiple hard drives. The second important variable is your workload, especially how dirty pages would line up on the hard drive. If there are a lot of sequential pages which are dirty Innodb will be able to use larger size IOs – up to 1MB flushing dirty pages which can be a lot faster than flushing data page by page.

So if we have system with single hard drive doing 200 IO/ssc, 48G buffer pool which is 90% dirty and completely random page writes we’ll look at 13500 seconds or about 5min per 1GB of Buffer pool size.
This is worse case scenario though it is quite common in practice to see shutdown time of about 1min per GB of buffer pool per hard drive.

Baron has written a nice post how to decrease innodb shutdown time which you may want to read on this topic.

Entry posted by peter | 7 comments

Add to: | | | |

10:15 A Global Greeting (990 Bytes) » Inside AdSense

Members of the AdSense team from all over the world say hello from Mountain View, CA!

Posted by Katrina Kurnit - Inside AdSense Team

07:26 Distributed Hashing Algorithms by Example: Consistent Hashing (826 Bytes) » High Scalability

Consistent Hashing is a specific implementation of hashing that is well suited for many of today’s web-scale load balancing problems. Specifically, it can be seen in use in various caching solutions like Memcached and is applicable to NoSQL solutions as well. Consistent Hashing is used particularly because it provides a solution for the typical “hashcode mod n” method of distributing keys across a series of servers. It does this by allowing servers to be added or removed without significantly upsetting the distribution of keys, nor does it require that all keys be rehashed to accommodate the change in the number of servers.

You can read the full store here.

03:15 Better than nothing (is harder than you think) (1430 Bytes) » Seth's Blog

Most of the time, particulary in b2b and luxury sales, the competition is nothing.

"I will buy this treat or I will buy nothing, because I don't really need anything."

"I will buy your consulting services, or I'll continue doing what I'm doing now on that front, which is nothing."

None of the above.

"I will vote for you or I'll do what I usually do, which is not vote."

"I'll hire you or I'll hire no one."

While you think your competition is that woman across town, it's probably apathy, sitting still, ignoring the problem... nothing.

Stop worrying so much about comparing yourself to every other possible competitor you can imagine and start comparing yourself to nothing. Are you really worth the hassle, the risk, the time, the money? Or can't the prospect just wait until tomorrow?

2010-09-01 Wed

13:45 Scale-out vs Scale-up (585 Bytes) » High Scalability

In this post I'll cover the difference between multi-core concurrency that is often referred to as Scale-Up and distributed computing that is often referred to as Scale-Out mode.

more..

Source: Scale-out vs Scale-up (http://www.dzone.com/links/r/scaleout_vs_scaleup.html) by Nati Shalom

12:15 Launching the ShipIt Workbook (2553 Bytes) » Seth's Blog

Six months ago, I put together a workbook that would help Linchpin readers ship.

After testing it out on hundreds of people, it's now ready for retail sale. [UPDATE on 9/2--yesterday, the workbook was so popular it went to the top 10 of all books on Amazon. And they sold all the warehouse could take. So it's sold out... I have shipped more to them, but they probably won't go on sale until the 8th. I'll update this post then. Thanks guys.]

You can find details here, or jump right to the buy page. The goal? To make you uncomfortable at the beginning of a project (and successful at the end).

Here's the core idea: it's weird to write in a book. When you do, you're making a commitment. You're combining the open-mindedness that reading brings with the physical action of writing. If you do that at every step in a project--and if your co-workers do too--the seemingly slippery decisions that get made appear a lot more solid.

The ShipIt workbook is designed to be worked on in groups (hence the five pack) and it delivers. If you can confront the mechanics or the fear that's slowing down (or even killing) your project, it's easy to fix it now, before it's too late.

There's no digital version, because without writing things down, it can't work. But there is an mp3 interview that will help you get your arms around how each page works. I'm pricing this first batch at $3.20 each in a pack of five just for the launch. [PS Amazon is having trouble shipping to Canadians right now. It may take a while to figure this out, and all I can do is apologize...]

I hope you'll give it a try.

10:40 Paper: The Case for Determinism in Database Systems (838 Bytes) » High Scalability

Can you have your ACID cake and eat your distributed database too? Yes explains Daniel Abadi, Assistant Professor of Computer Science at Yale University, in an epic post, The problems with ACID, and how to fix them without going NoSQL, coauthored with Alexander Thomson, on their paper The Case for Determinism in Database Systems. We've already seen VoltDB offer the best of both worlds, this sounds like a completely different approach.

The solution, they propose, is:

03:45 Responsibility and authority » Seth's Blog