The intersection of Technology, Leadership and Society that made Aadhar happen.A captivating account

 My review at Amazon, of the book "The Aadhar Effect", written by my friend and inspiration, Ramnath.

Technology work with Indian Government is  hard. Harder still, when private and public institutions have to work together. The hardest part is when it involves each of the 1 billion citizens as an end user. The Aadhar project faced all these levels of hardships. The book, 'The Aadhar Effect' seeks to capture this hard, complicated and multi-faceted journey to consolidate and de-duplicate the Indian digital identity.

Somewhere in the book, the authors mention that the project had many aspects of a thriller with twists and turns, conflicting forces, drama etc. Well, the book itself seems to be that! Particularly the sections on privacy, with RTI activists .vs. Privacy activists, are page turners. The book is beautifully written. At many points, when I was reading a section, I had some thoughts (ex: about fortune at the bottom of the pyramid) and as I progressed, I discovered they were exactly covered in the book. That shows a natural flow that evokes reading interest.

The book goes beyond Aadhar itself and discusses enabling digital platforms and digital services that are or could be built over Aadhar. There is a small amount of noticeable (though well-deserved) Nandan praise for his leadership abilities, his ability to conceive the larger aspects of the project and put a diverse team together. To be fair, in the chapter 'Who's afraid of Aadhar?', the book lists a collection of 50 common problems and objections to Aadhar. Nowhere else can you find such a comprehensive counter-view in one place.

At some places, the authors tend to delve on the 'larger picture', philosophically or strategically, which may be viewed as a stretch. Like, towards the end, they portray Aadhar like it's some foreign relations enabler for its learning potential for implementing large digital projects.

The angle on Aadhar being, an important, but only one of the many Lego blocks is a significant theme of the book. In the same lines, the book dwells upon future possibilities and great potential of Aadhar, rather than on how it has been useful in the development arena, the exceptions being the Direct Bank Transfer scheme and payment systems. It dwells on the architectural beauty of having a unique identity and how it can be a true transformation tool if you want to use it for development and empowerment and how the central payments infrastructure is already making it possible.

The book has a good share of the stories of getting work done. For example, with Nandan's backing by the PMO, his access to PM worried the bureaucrats,. The authors mention it as: 'It's not that he would call. It's that he *could* call'. The book is of full of such super-interesting tidbits and also many anecdotes about the dynamics between Nandan Nilekani, his colleagues and various government departments.

As you write a book review, you realize that you are not writing a review of the book, but a review of the Aadhar project itself! Thats a good thing, it means the book has achieved its goal, of pushing you to think about the project, its impact, potential and ramifications.

If you seriously think about the impact of technology in Samaj and Sarkar, if you tend to have an 'integration' view of all things tech and the power of digital platforms to transform landscapes, this book is a great read. If you have been in situations, where you have to demonstrate extraordinary leadership to bring a diverse team together to push towards a visionary goal, you must pick up this book. You will understand both, how hard it is do good work in India and how hard it isn't, i.e. if you have the commitment, the right team and undaunted persistence.

It may be waiting for you at the airport, at a book fair, at a neighbourhood store, or in Amazon. It might pick you up. Like they say, when it comes to good books, you don't choose them, they choose you.

Teaching Dinosaurs to dance digitally - Part 2

Part 1 of this article was about why PDF India is not #DigitalIndia and some thoughts on access to public data and data submission. 

Now that the rant is over, let me document some of the woes that I came face to face in accessing public data.

A couple of years back, I was consulted by a friend once on having to extract select data fields from hundreds of PDFs from SEBI website, for research work. It involved the extraction of fields such as Offer Price, Negotiated price etc (well, whatever their finance lingo), from Letter of Offer documents submitted by merger/takeover companies to SEBI, over 15 years. Putting together a strategy using open-source/free tools and an eye to automate most of the tedious tasks to the maximum and maintain accuracy, a sequence of steps was devised for automated extraction. In some stages, semi-automated. Some steps required human eye reviews from time to time, about whether the automation ghost, that eerily moved the mouse pointer at midnight, is working properly or is being interrupted by the invalid data ghost.

Here are some of the concerns I came across. 

The SEBI page of Letter of Offer for Takeovers, was treated as the starting point for collection of data fields related to Final Letters of Offer. ​

Original data collection by SEBI is not structured :

The fact that information was collected as PDF and not as structured data gives less scope for meaningful analysis of data. Ideally, the data should have been collected from the companies in more organized formats such as XML, Excel or CSV or by seeking information in a web form, as is the standard practice. This would have allowed the collection of raw data by SEBI in a database-friendly format. Since this was not to be, it led to a situation where the structured data have to be sought by sifting through PDF documents. For ages, we have been looking at standards like IFRS and XBRL from a distance, but nothing moved because SMEs complained of compliance cost.

No options to bulk download :

There were no options to bulk-download documents by querying for multiple companies based on search criteria. One had to traverse the pages a few links at a time to download the documents. This constraint was later overcome partially by a series of semi-automated steps using tools such as FlashGot, TinyTask and the parsing of the pages for file paths.

Inconsistency and non-standard methods in the organising of PDF links in the SEBI website :

The formats in which the document links were organized varied between pre-2005 and post-2005 periods. Links to pre-2005 takeover documents, would lead to a direct download link of the Final Letter of Offer. The file name would be numeric, giving no idea about the company in question. Such as this.  On the other hand, post-2005 takover documents, would lead to another intermediary web page of links for the company (such as this), which in turn would lead to the PDF link. In the later years, the PDF file would be named more meaningfully such as LOF etc (which is a relief), but not consistently. This meant more manual downloading and filtering of necessary documents from unnecessary ones. Some HTML tags filtering using tools such as NotePad++, followed by exporting to a database, were used to partially overcome these constraints.

Unsuitability of PDF format for structured data :

The PDF format doesn't lend itself to efficient parsing to collect data. Moreover, the data fields required (such as Average price), were often presented in a table inside the document. This meant that the PDF documents needed to go through a series of steps before data could be culled out from them. Calibre software and the online service pdfonline.com were used in batch processing to convert PDFs to HTML web pages. The individual code lines from the web pages were exported to an SQL database and parsed for HTML tags to look for tables that contained the required fields.

Lack of uniformity and fixed format for the presence of data in the document :

Inside the PDF document, data was not found consistently in the same location or format. One had to look for references to the section title such as 'Financial Justification' and then look for the data in the table that geographically followed it below. Keywords such as 'Negotiated Price', 'Average Price' and various combinations of such phrases had to be listed to look for data. Even after all this, one wasn't sure whether the data fields would indeed be found or not. 

Inefficiency of using text-based search to identify data fieds :

When found, they may not be found in a consistent form or a phrase. For instance, 'Not Applicable' could be any of NA, N.A., N-A,a hiphen etc. Sometimes, the justification section would be a paragraph that contained none of the keywords. For an automated lookup, references to 'infrequently traded' might often been confused with 'infrequently traded on NSE and frequently traded on BSE'.   These made automated collection inefficient and necessitated a manual study of the PDF documents.

Mismatch between different databases in company name mentions :

While some data fields required for the study were to be found from Letters of Offer, others were extracted from the Prowess database. The takeovers involve an acquiring company and a target company. There were inconsistencies and mixups in both these names between the Prowess and SEBI data sources. SEBI would list the letter of offer against the target company whereas Prowess would list it the other way round. In some cases, where there were multiple takeover instances relating to same company names, requiring the PDFs of multiple years. This meant resolving the mix-up by manually reviewing the PDF and data fields. One had to arrive at a cross-tab of SEBI name, Prowess Name and the correct Letter-of-Offer-Name in the takeover context before one could collect accurate data.

---------------------------

This is not the issue of a single website of one government insitution or a government department. It’s the opposite, of being unable to work as a single website for the whole country. It’s about the isolated approach to.presenting data to the public or seeking data from them, sulking it as an unwanted duty imposed by the Digital India frenzy.

The data.gov.in  sets an excellent precedent with public APIs throwing an ocean of private opportunities open. It is high time that other government institutions join the bandwagon of unified data view architecture both for submission and presentation of data, in truly seamless transactional digitization, keeping in view bulk submission and bulk download options. Data.gov.in has plenty of uncompleted agenda ahead of having to unify the data from the states at a regular timeline and keeping it update and in getting the various uncomplying departments to co-operate.

Some data dinosaurs have to be taught to dance digitally, because evolution is binary. 
You either become distinct or you become extinct. 

Teaching dinosaurs to dance digitally

It ain’t so much of #DigitalIndia yet. It's just PDF India. For those looking for meaningful, processible data, the difference between the two can make a world of difference. For some government institutions, digital means PDF or JPEG image of scanned print reports, including spreadsheets and balance sheets. One might argue that nowadays it's possible to extract data from PDFs using technology. But it's like garbage disposal and recycling. The messup shouldn't be created in the first place. Image processing algorithms were not invented for spreadsheets.

A good example of a bad PDF document is the District Census Handbook of 2011. It will have reams and reams of pages, with one page showing the first half of a horizontal spreadsheet, titled “Industrial Catgory of”, and next page showing the other half, titled “Marginal Workers”. Man, that must be some innovation in bad design .

Switch to the good example of presentation of the same data. data.gov.in presents hundreds of datasets from the Census of 2011 in open formats for public access. You can access the data for the same district in CSV format, loadable into Excel. Did you know you could download the Indian Railways timetable in Excel ?. How sweet!

The salient rule in data collection or presentation must be that, at the raw source, the data must be collected in a format that is processible by automation. It should minimize human eye intervention only for reviews and green/red flags and throwing up exception patterns. Or for discovering insights of wisdom from "rich experience", which a chip can't discern. Days may have come when chips outsmart elders in experiential wisdom as well, LoL.

Companies like HowIndiaLives.com , where my friend Ramnath is involved, address this problem by helping their customers and website visitors make visual sense out of the non-sense data in India's public domain. They also present it in a beautiful and user-friendly ways and help project stakeholders glean useful insights from data. That data shouldn't have been nonsensical in the first place, in its raw open form, is the sad fact. That they converted rants like these into a business opportunity is their ingenuity. While raw data must be disseminated in open processible formats, it should lead to an ecosystem of companies like these, which compete in discovering insights from data and presenting them, without having to spend too much time cleaning it. Not just cleaning it, but having to fight inconsistencies between multiple sources of data for the same item of information, I guess, must be another tedious task.

Taking a leaf out of international open data like data.un.org, the open data platform for India is a big leap in this direction for sharing of public datasets (though they don’t have options to bulk download data). 

On the other hand, some of the transactional websites of the government websites can make life extremely tedious. If you are a high-volume transaction submitter, your life can become miserable, having to submit thousands of records into old-style web forms. Some of them must have been in a cave since AJAX was invented. They can put thousands of person names with option buttons on a single page, expecting the user to scroll down or use the browser's Find, choose one name and then submit. Wwwhaaat!

The format in which data must be submitted to government websites must be pre-defined with digital processing in mind. For a billion-headed Titanosaur like India, it should definitely have scale in mind too. Ideally, it shouldn't even be Windows OS intensive and Windows OS requiring. Kerala, for example, wants to dabble in Linux and it's a good thing. I hope they don't up give up like Munich, the city that wanted to run on Linux. But, that kind of thinking is good. It may lead at least to the adoption of open formats for seeking data, if not an open-source OS.

Not all is a sad story with Digital India. There are a handful of bright spots in good design, that take scale and digital processibility into account. Aadhaar, no doubt, is a beautiful example. The Income Tax website often has some sudden quirky differences between its Java tool and the Excel tool with mysterious conclusions of inability to generate the XML file. But it at least uses XML to upload data. Thats a good thing. Even within the Income Tax Department, you may not find the same kind of good design for other tasks, for instance, for applying for non-deduction of TDS. Another of my favorite examples of handling technology at scale carefully is SBI's transition to core banking and their merger with associate banks. It was not about open data or about government per se. But, at its scale, it's truly a project of teaching elephants to dance, and for their size, they did a mighty good job at it. The GSTN must be the next Aadhaar-like unifier, after the easing out of the initial troubles.


This is the intermission, the end of Part 1. 

In Part 2, I mention some of the woes I faced while extracting data from PDFs from the SEBI website. It has a sad ending that rounds up by saying :

Some data dinosaurs have to be taught to dance digitally, because evolution is binary. 
You either become distinct or you become extinct.  

Lot of technical debris ahead on Part 2. Blissful poets, musicians and other non-tech readers not allowed beyond this point. :-) :-)

-->> Part 2 

What's your Personal Work Ethic ?

The personal work ethic is a topic I always wanted to write about. Ramnath triggered this piece, by posting a short review of the book, The 4-hour Workweek at the Sai Students Portal. The book is mainly about how you can manage to work only 4 hours a week and amongst others, suggests outsourcing personal tasks. Some discussion ensued and here are my comments at that blog, made out into a post (with a few edits ) here :

I havent read the book, I find the title and theme of the book as described, quite interesting. But I find that the "methods" that he suggests are a bit cliched, just a e-Yuga rehash of the old school lessons of time management, personal efficiency, goal setting stuff talked by a lot of other books.

That apart, the ability to contemplate on why we do what we do and the conflict between what we want (at our ideal level of aspiration) and what we do, is something we lack in our times. What Dritharashtra said in the context of Dharma is also applicable to goals, Jaanami Dharmam Nacha Me Pravritti, Jaanami Adharmam Nacha Me Nivritti..., the gap between knowing what to do and doing it in-deed.

The personal work ethic that each of us bring to the workplace is something that I have always found interesting to observe. How much of what we do is because of the control system that pushes us and how much of it would we do in its absence ? How many hours of work is "right" or "optimal", assuming you want to be just loyal to the contract, not any less or more ? Peter Drucker once said, that the best motivated employee is a volunteer. What is the substantially differentiating basic attitude towards work, between, say a waiter whose opportunity to bring originality to the work is limited, and say, a Google employee who gets to spend 20% on it ?

Why work ? Well, that can be a dangerous question :) . If you deconstruct this too much from the Advaitic angle, you might end up with a fallacious conclusion : Don't work. Which we knew already and we are good at. :) To avoid that, you should start from Gita's premise, that work is inevitable. You can't not work, dude, the software doesnt provide that feature. Given that, and all of us have the same 24 hours, a deep thought on 'Why do I work ? ' can provide custom answers to what satisfies you. It can differentiate achiever from a non-starter, a poet from a mechanic, a saint from a sinner. It can be a basis for the development (or lack thereof) of other qualities like loyalty, dedication, team spirit and ambition. You can find some of these qualities and an amazing work ethic in some workers and simple people, so it may not actually be a function of the money you get. It's probably just a function of what custom answers you form in your mind, after you solve the equation for yourself.

Of course, there are major implications when you institutionalize the optimal work ethic of an individual, because, "market" forces like competition, cost, performance pressure, peer presence etc chip in to make it complex. But probably, the large scale orientation towards metrics and efficiency in modern management, is pushing the individual more and more away from his or her optimal band of work-life balance. As a race, we have moved from success in survival to success in war to success in trade. The common element in those phases has been competition. What's the next dimension of success we will move towards ? And what will that revolve around ?

I also wonder whether people doing one kind of work (say intellectually challenging strategy work ) are any more "busy" than, say, a construction worker who carries bricks all day. We usually think the former kind to be much more "busy" and perhaps "better contributive", "better value-adding" than the latter, but, in terms of the time spent, they both spend the same amount of time (give or take a few hours) on something that they have chosen (or say forced to have chosen) to do. And in most cases as part of a contract. When someone says, I am more busy than you, it's most often untrue, it just means, what I am busy with, has more visibility than what you are busy with. Or, I may have all the time in the world, but that time is not for you. :)

One type of work may be more satisfying than another, depending on what satisfies you. But is one type of work, intrinsically superior/good than another ? Is a painter better off than a conductor, because his work is creative ? If it is, what parameters contribute to its superiority ? Say, "to create a better world", is one such. The person who is at the top of such a company identifies directly with it and probably closer to that vision whereas for the person who is involved three levels down the work hierarchy, it would just be, being a waiter, a job to do for the pay he takes. The reverse is also possible in their attitudes, someone takes to it as carrying a stone (or pushing numbers), and someone else takes to it as building a cathedral. We once spent a whole night loading trucks with rice, clothes and relief material for the victims of the Gujarat earthquake. Oh, we found it very satisfying when the series of trucks were leaving the campus. Why was that ?

I think there are actually very few templates in which majority of us fall in. Very few actually get to do something that is substantially different, creating a new template altogether. Although we often want to claim and feel what we do is somehow "unique", and say so in our marketing brochures and interviews, most of the time it's the same cycle and the same pursuits, with minor variants/derivatives of what we call in programming design as an Abstract Class. Brings me to the thought: how much of programming work is different from plumbing ? You fix one valve and there goes the next, phut. What, we actually use the words like architecture, platform, address, tunnelling and named pipes... :) If you push us a little more, we'll start coming out with software equivalent terms for concrete, steel, emulsion paint, waterproofing and so on.

In every area of work, there is the exciting part, the boring part and the hated part. That exciting, boring and hated tasks come as a package in any vocation is something you may have to live with. Like doing the dishes after the party. For this reason, I have always failed to resolve one of the usual guidelines that personal effectiveness books suggest : Prioritize and ensure you give your time to high value-adding tasks. The fallout of this is that you are forced to categorize a certain set of tasks as low priority, with the effect that they are first ones to get rescheduled or postponed. Over a period of time, these tasks will build up to become critical or requiring immediate attention and graduate to become high priority and then you run to it. Whereas the very buildup should have been averted in the first place if you paid due attention to those seemingly low priority items on a more distributed basis. Cleaning, maintenance, fixing things that dont work, backing up your computer, stitching that button in time and a hundred other little things would be called low priority in a "Value-Time-Matrix" that these books would draw for you. Of course, the rationale is to avoid getting lost in a ocean of little things, but too much focus on high value items only results in escalation purely born out of negligence. At the workplace, everyone tending to high value prioritization can cause defects that are not noticed and people finding their own little ways to cut corners even as they continue to present a nice greeny picture on the high value items.

To be able to give every task its due entails the acceptance of a certain amount of boredom that comes as a package with enjoyable work. Tenacity, thoroughness, exhaustive level of detailing can all turn to boredom, but one may have to go beyond that attitudinal fatigue to be able to deliver good work.

On a personal level, outsourcing comes at a more pinching cost than it does for orgs. In the hostel, there were always two sets of guys, who always washed and pressed their clothes even if they could afford and the ones who outsourced them. I used to find it strange when someone said "I love doing this myself". My favorite outsourcing question used to be : why dont you grow your own paddy ? :) . Hmmm, that explains the success of Farmville ! (and my failure therein).

The famous quote "enjoy what you do and you don't have to work a day in your life" is so cliche now. That also entails a certain amount of re-orienting our attitude towards work, if not opting out of the entire work stream that one may be currently involved in. I think it's a flowery way to encourage those who find that boring and hated components of the package are greater than the exciting part. Either find work that you enjoy or learn to "book" something as enjoyable :) :)

This shouldn't be depressing, however. The point is that, what we think about the work we do and want to do, has a lot to do with how happy we are. Happiness, in a mundane sense, is a function of what we do, why we do and how we do but it's like an ice cream. As long as you get the flavour A you like, you are as averagely happy as another person who liked and got his flavour B.

The ideal personal work ethic would probably be a cross-product of buddhist, protestant and the Gita approaches to work, with collectivist lessons drawn from communism and the achievement orientation drawn from capitalism. Well, that would be NextGen Sociology !!

Blogmentor Parampara

So... Ladies and Gentlemen, I am here, finally. Been thinking of starting a blog for the past two years but it never materialised for the usual reasons. One of these Web 2.0 sites should study what inhibits people like me with an initial inclination to blog/interact on the web and makes them put it off for later. And probably also why interest wanes after a while.

Among others, the two things that helped it happen are :

(a) Inspiration from a friend whom I should call my "blogmentor" Ramnath
(http://ramz.blogspot.com/ ) and
(b) a laptop, broadband connection in my room.

I was never comfortable about the idea of blogging from office and I am too lazy to walk down all the way to a cyber cafe for the noble cause of creative writing.

Two things to create when you need to make people buy a product/service:
1. Interest and
2. Access. (Aaha, what gyaan !!) .

There should be something like the blogmentor parampara like the guru-shishya parampara, who inspired whom to start blogging.

Started the blog on the Vijayadasami day, but quite funny that I didn't know what to write after I created the blog. So today I thought I will push myself to write something up. To call it a writer's block will be too hi-fi a term to describe the starting trouble. Nor can I say my mind went blank, you might think I have attained too high a spiritual state too quickly. Probably I should start by providing links to stuff I have written earlier but that would be like serving cold bindi fry from the refridgerator, the majaa is lost, not on Day 1, so will reserve that for a gloomy day. Hopefully, I will pick up some momentum and write without pushing.

In the meanwhile, you are welcome to push me. And thanks Ramnath for the first.