2010-09-01 Wed

13:45 Scale-out vs Scale-up (585 Bytes) » High Scalability

In this post I'll cover the difference between multi-core concurrency that is often referred to as Scale-Up and distributed computing that is often referred to as Scale-Out mode.

more..

Source: Scale-out vs Scale-up (http://www.dzone.com/links/r/scaleout_vs_scaleup.html) by Nati Shalom

12:15 Launching the ShipIt Workbook (2241 Bytes) » Seth's Blog

Six months ago, I put together a workbook that would help Linchpin readers ship.

After testing it out on hundreds of people, it's now ready for retail sale.

You can find details here, or jump right to the buy page. The goal? To make you uncomfortable at the beginning of a project (and successful at the end).

Here's the core idea: it's weird to write in a book. When you do, you're making a commitment. You're combining the open-mindedness that reading brings with the physical action of writing. If you do that at every step in a project--and if your co-workers do too--the seemingly slippery decisions that get made appear a lot more solid.

The ShipIt workbook is designed to be worked on in groups (hence the five pack) and it delivers. If you can confront the mechanics or the fear that's slowing down (or even killing) your project, it's easy to fix it now, before it's too late.

There's no digital version, because without writing things down, it can't work. But there is an mp3 interview that will help you get your arms around how each page works. I'm pricing this first batch at $3.20 each in a pack of five just for the launch. [PS Amazon is having trouble shipping to Canadians right now. It may take a while to figure this out, and all I can do is apologize...]

I hope you'll give it a try.

10:40 Paper: The Case for Determinism in Database Systems (838 Bytes) » High Scalability

Can you have your ACID cake and eat your distributed database too? Yes explains Daniel Abadi, Assistant Professor of Computer Science at Yale University, in an epic post, The problems with ACID, and how to fix them without going NoSQL, coauthored with Alexander Thomson, on their paper The Case for Determinism in Database Systems. We've already seen VoltDB offer the best of both worlds, this sounds like a completely different approach.

The solution, they propose, is:

03:45 Responsibility and authority (851 Bytes) » Seth's Blog

Many people struggle at work because they want more authority.

It turns out you can get a lot done if you just take more responsibility instead. It's often offered, rarely taken.

(And you can get even more done if you give away credit, relentlessly).

2010-08-31 Tue

17:52 Introducing tcprstat, a TCP response time tool (6305 Bytes) » MySQL Performance Blog

Ignacio Nin and I (mostly Ignacio) have worked together to create tcprstat[1], a new tool that times TCP requests and prints out statistics on them. The output looks somewhat like vmstat or iostat, but we’ve chosen the statistics carefully so you can compute meaningful things about your TCP traffic.

What is this good for? In a nutshell, it is a lightweight way to measure response times on a server such as a database, memcached, Apache, and so on. You can use this information for historical metrics, capacity planning, troubleshooting, and monitoring to name just a few.

The tcprstat tool itself is a means of gathering raw statistics, which are suitable for storing and manipulating with other programs and scripts. By default, tcprstat works just like vmstat: it runs once, prints out a line, and exits. You’ll probably want to tell it to run forever, and continue to print out more lines. Each line contains a timestamp and information about the response time of the requests within that time period. Here “response time” means, for a given TCP connection, the time elapsed from the last inbound packet until the first outbound packet. For many simple protocols such as HTTP and MySQL, this is the moral equivalent of a query’s response time.

The statistics we chose to output by default are the count, median, average, min, max, and standard deviation of the response times, in microseconds. These are repeated for the 95th and 99th percentiles as well. Other metrics are also available. Here’s a sample:

[root@server] # tcprstat -p 3306 -n 0 -t 1
timestamp	count	max	min	avg	med	stddev	95_max	95_avg	95_std	99_max	99_avg	99_std
1276827985	1341	24556	23	149	59	767	310	91	69	1030	107	112
1276827986	1329	12098	28	134	63	461	299	91	65	667	104	93
1276827987	1180	13277	22	202	93	873	439	103	79	1523	131	169
1276827988	1441	15878	27	180	139	672	427	116	79	1045	136	128
1276827989	1432	157198	26	272	138	4165	405	115	80	1092	134	123
1276827990	1835	25198	26	183	124	734	448	115	85	1141	137	141
1276827991	1242	6949	29	129	114	301	233	98	61	686	109	84
1276827992	1480	284181	25	442	127	7432	701	128	114	4157	173	293
1276827993	1448	9339	22	161	88	425	392	104	80	1280	126	140

tcprstat uses libpcap to capture traffic. It’s a threaded application that does the minimum possible work and uses efficient data structures. Your feedback on the kernel/userland exchange overhead caused by the packet sniffing would be very appreciated — libpcap allows the user to tune this exchange, so if you have suggestions on how to improve it, that’s great.

We build statically linked binaries with the preferred version of libpcap, which means there are no dependencies. You can just run the tool. In the future, packages in the Percona repositories will provide another means for rapid installation via yum and apt.

tcprstat is beta software. Several C/C++ experts reviewed its code and gave it a thumbs-up, so many eyes have been on the code. We’ve performed tests on servers with high loads and observed minimal resource consumption. I personally have been running it for many weeks on some production servers without stopping it and have seen no problems, so I am pretty sure it has no memory leaks or other problems. Nevertheless, it’s a first prototype release, and we want much more testing. We might also change the functionality; as we build tools around it, we discover new things that might be useful. When we’re happy with it and you’re happy with it, we’ll take the Beta label away and make it GA.

The tcprstat user’s manual and links to downloads are on the Percona wiki. Commercial support and services are provided by Percona. Bug reports, feature requests, etc should go to the Launchpad project linked from the user’s manual. General discussion is welcome on the Google Group also linked from the user’s manual.

[1] Historical note: we initially called this tool rtime, but did not publicize it. However, some of you might have heard of “rtime” before. This is the same tool.

Entry posted by Baron Schwartz | 6 comments

Add to: | | | |

11:15 Just launched: Linchpin on the Vook on the iPad (2062 Bytes) » Seth's Blog

The details are right here. Created by Vook, based on the hardcover.

Includes new video and interviews with some interesting folks...

The long tail challenge of the iPad store is getting more and more obvious to people. The ratio of "shelf space" to inventory is about the worst of any retail experience in the world. There are more than 24,000 apps listed in the iPad store, and yet the front window (equivalent to the window of a bookstore) shows the user six choices. The spotlight coverflow up top shows another sixteen, fairly randomly. Meaning there's a little worse than a one in a thousand chance that your app will appear in front of someone interacting with the store at the first level.

I have no doubt that as Apple sees revenue increase from this source, they'll do a much better job of crosslinks and browsing. But, once again, the lesson of the long tail is this: you can't count on the gatekeeper to do your promotion for you. Getting picked feels like a needle in a haystack, and the value of permission, of connecting directly to people who care instead of ceding control to a middle man, is at the heart of building an asset. Someone is going to be the gatekeeper, and it should be you.

03:15 The corporate conscience (1681 Bytes) » Seth's Blog

There isn't one.

Corporations don't have a conscience, people do.

That means that every time you say, "It's just my job," or "My department has a policy," or "All I do is work here," what you've done is abdicated responsibility--to no one.

It's convenient and even comfortable to blame the anonymous actions of many working in concert on a evanescent brand or organization, but that starts you on an inevitable race to the bottom. Organizations have more power than ever before. They are better synchronized, faster, and possess more tools to change the economy and the people in it than ever before. And the only option available to the rest of us is for individuals to take responsibility (it's not given) for what they do and how they do it.

The very same tools that permit organizations to synchronize their efforts are now available to you and to me. I guess the question is: will we use that power to humanize the systems we've created?

PS It's not just about being a good citizen: when bad behavior comes back to hurt the company, it hurts you, too.

2010-08-30 Mon

09:15 AdSense email preferences: Get the most from your account and from Google (2829 Bytes) » Inside AdSense

Have you checked your AdSense email preferences lately? If not, you may be missing out on important information and special offers without realizing. Take a minute to log in and update your email preferences under the ‘My Account’ tab.

Want to be invited to upcoming events like AdSense In Your City? Make sure you check the box next to Special Offers when you edit your email preferences so you can stay up-to-date with giveaways and other special programs in your area.

Want tips from the AdSense team for how to earn more with your AdSense account? Check Customized help and performance suggestions and/or Newsletters so we can offer personalized guidance to improve performance and maximize your revenue.

Want to help us improve AdSense by testing out features like the new AdSense interface? Check Google Market Research and you’ll be able to share your valuable feedback with us through surveys and beta tests.

Want promotions and key updates for other Google products that can help you grow your AdSense business? Check Information about other Google products and services which may be of interest to you so we can send you news and coupons based on your potential needs.

We want to help you earn more with tips and promotions, invite you to in-person events, and gather your feedback so we can continue to improve AdSense. So log in, update your preferences, and take advantage of the opportunities coming your way from the AdSense team!

Posted by Katrina Kurnit - Inside AdSense Team

07:03 Pomegranate - Storing Billions and Billions of Tiny Little Files (2143 Bytes) » High Scalability

Pomegranate is a novel distributed file system built over distributed tabular storage that acts an awful lot like a NoSQL system. It's targeted at increasing the performance of tiny object access in order to support applications like online photo and micro-blog services, which require high concurrency, high throughput, and low latency. Their tests seem to indicate it works:

We have demonstrate that file system over tabular storage performs well for highly concurrent access. In our test cluster, we observed linearly increased more than 100,000 aggregate read and write requests served per second (RPS).

Rather than sitting atop the file system like almost every other K-V store, Pomegranate is baked into file system. The idea is that the file system API is common to every platform so it wouldn't require a separate API to use. Every application could use it out of the box.

The features of Pomegranate are:

It handles billions of small files efficiently, even in one directory;
It provide separate and scalable caching layer, which can be snapshot-able;
The storage layer uses log structured store to absorb small file writes to utilize the disk bandwidth;
Build a global namespace for both small files and large files;
Columnar storage to exploit temporal and spatial locality;
Distributed extendible hash to index metadata;
Snapshot-able and reconfigurable caching to increase parallelism and tolerant failures;
Pomegranate should be the first file system that is built over tabular storage, and the building experience should be worthy for file system community.

Can Ma, who leads the research on Pomegranate, was kind enough to agree to a short interview.

03:15 Professionals, amateurs and the great unwashed (1279 Bytes) » Seth's Blog

If you want something done, perhaps you would ask a professional to do it. Someone who costs a lot but is worth more than they charge. Someone who shows up even when she doesn't feel like it. Someone who stands behind her work, gets better over time and is quite serious indeed about the transaction.

Or perhaps you could hire a passionate amateur. That's a forum leader doing it for love, not money. An obsessive in love with the craft. A talented person willing to trade income for the chance to do what he loves, with freedom.

Please, though, don't hire someone who just thinks it's a job. This category represents the majority of your options, and this category is what gives work a bad name.