Publicly tracking how a government spends its money is more than just posting documents online – making sense of budgets is crucial to providing oversight and interpretation of government spending. Working to demystify this process is at the root of the Open Budgets India project, which is fighting for a more free and open approach to budgets in India.

A beta project with 5,300 datasets from 333 sources, the portal provides access to budgets at every level of government in India, providing APIs and visualizations for previously obscure data. The project also created a useful FAQ to help people understand how government budget data works, how it differs from other kinds of data, and why it matters. Funded by a number of institutional partners, the portal has served more than 3,000 users looking for visualizations and information on open budgets at local and federal levels.

Why open budgets? Why are open budgets important to open government?

Government budgets are a comprehensive statement of government finances for a financial year, translating government’s promises and priorities as expenditures and its receipts to meet such demand. Open Budgets Data is government budget data that is publicly accessible (uploaded online on a timely basis) in a machine-readable and reusable format covering all data points (not just analysis), freely available and legally open to use for everyone without any restriction. Over the years, open budgets data has become vital to build trust in government’s financial activities and sustaining transparency in its policy decisions. It also enables citizens, policy makers, civil societies, journalists, and other key players to engage in budgetary processes and strengthen participation and insight into budgetary policies in the country.

In order to have open and effective governance, governments need to invest and commit on complete budget transparency, i.e. full disclosure of open budgets data on revenues, allocations and expenditures across the public sector. Unfortunately, in India public access to budgets diminishes as we go deeper from the level of the Union Government (Central Government) to the subnational level. As a result, the use and analysis of budget data has been restricted, and the scope for citizen’s engagement with government budgets has been limited. In such a backdrop, Open Budgets India (OBI), a comprehensive and user-friendly open data portal, facilitates free, easy and timely access to relevant data on government budgets in India.

Data visualization: Outstanding External Debt from Open Budgets India

What are the greatest successes and obstacles you’ve faced with this project? How have you seen it used so far?

From the very inception of the project, we have collaborated with a diverse team of researchers, technologists, data scientists, policy hackers and groups of volunteers to co-create this portal. We have embraced open-source technology, design, visualizations and documentation to go several steps ahead and become an open-source initiative, facilitating transparent and accessible co-development. We have automated conversion of budget PDFs into clean CSVs, created time-series data and developed scalable visualizations. This has helped us to scale our data mining techniques across various tiers of governments. Also, we forked CKAN, an open-source data portal platform and customized it as per our needs. Use of open-source softwares has reduced our development time manifolds thus we have kept all our work in open too, so that other organizations can freely reuse it and may help us to make better systems. Also, with the help of our automated data pipeline, we were able to publish machine readable budget data from Union Budget 2017-18 in less than 24 hours, enabling timely and informed budget analysis.

One of the major obstacles towards budget transparency in India is a lack of consistency and standardization in budget data formats across years and government bodies. Few states still avoid publishing their budget documents online and other have fonts and character encoding issues, making it difficult for us to parse those. When it comes to disaggregated detailed data for State Expenditure and Receipts, format varies drastically as no two states follow a similar format for publishing their documents. Some of the budget documents are available only in local languages, thus requiring efforts in translation for interoperability. These issues make it difficult for us to produce crucial machine readable data for state budgets. However, we have developed a technique to generate CSVs for Karnataka and Sikkim, which we plan to scale up for other states in near future.

Unavailability of budget documents online and heterogeneity of the formats further increases as we move down to districts and municipalities. We even need to file RTIs (Right to Information) to several government bodies to acquire these important budget documents. Publishing of such information in a timely and accessible manner can strengthen monitoring of public expenditure as well as engagement of people with budgetary processes. That, in turn, can lead to significant improvements in the manner in which such allocations are spent.

In last two months, more than 3K unique users have visited the portal, with highest traffic on data visualizations followed by state and union budget documents. Users are finding CSVs and time-series datasets useful for doing their own budget analysis. It is also encouraging for us to know that other data portals and communities are using Open Budgets India as a source for budget data. For example, the urban data portal, OpenCity.in has credited OBI as a data source for couple of municipal corporation budgets.

Data sectors on Open Budgets India

How have governments reacted to having their budgets online and open? What kinds of responses how you seen from various departments?

On 27th January 2017, we conducted a public consultation on ‘Opening Up Access to Budget Data in India’ in New Delhi. The consultation included a panel discussion with experts on what should government authorities and civil society organisations pursue, in the coming years, so as to ensure that people get free, easy and timely access to relevant budget data at various levels of government. Sumit Bose, member, Expenditure Management Commission and former Finance Secretary, advised the project, saying, “A lot of hard work must have gone into developing this project, as budget preparation itself is a humongous exercise. Besides, experts, the volunteers need to be recognized, as they must have contributed in a big way in developing this excellent portal. I’m optimistic about this project… However, I feel that there’s no deliberate attempt by state governments, barring exceptions, not to put machine readable documents online. Probably, they never felt the need or it simply didn’t occur. In the next stage, the CBGA needs to tell state governments about their requirement.”

Deputy Comptroller and Auditor General, K Ganga suggested a few guidelines to strengthen the demand for making budget data available in public domain as well. She said, “Transparency can only be achieved once the common man outside of academics and governance understands budget data and use it. Things are done by keeping only the experts in mind. Besides, communication channels with every stakeholder be kept open even through vernacular medium, ensure that people can access data in various kinds of devices, create an environment by encouraging people to provide data, and pester the government to share data and information.”

Your project aims for accessibility and ease of use within government data. How are you working to make that data accessible despite a variety of file types and standards across organizations?

One of the major additions is a comprehensive metadata for budget datasets, which drastically increases searchability of documents. We have also classified datasets by tiers of government, developmental sectors(like Agriculture and Allied Activities, Health, Education, etc.) and data formats. Apart from adding machine readable files(CSVs), we have also created a number of time-series datasets enabling users to comprehend various trends in budgetary allocations across fiscal years. For Municipal Corporations, we have developed an unified format called as Budget Summary Statement to produce aggregated figures comparable across the municipalities. For each dataset, we have clubbed all the available file formats i.e. CSV, XLS and PDF as multiple resources in a single package so that users can preview and download the format of their choice. We also provide an API to programmatically access all 5.1K+ datasets from our portal, this enables developers to automate their search and download processes.

Are you accepting contributors? If so, how could people get involved?

Open Budgets India has resulted from collective efforts by many organisations and individuals, led by Centre for Budget and Governance Accountability (CBGA). Centre for Internet and Society (CIS), DataMeet, DataKind – Bangalore Chapter and Omdiyar Network (ON) have helped significantly in the conceptualisation of the project. A team of pro-bono data scientists led by DataKind Bangalore has helped us develop few key components of the portal. Macromoney Research Initiatives has helped us in making available budget data of a large number of Municipal Corporations. Our partner organisations across the country focusing on governments budgets, viz. Budget Analysis Rajasthan Centre (BARC), Jaipur; National Centre for Advocacy Studies (NCAS), Pune; and Pathey, Ahmedabad, have contributed their efforts in collecting, collating and translating budget data of a number of Municipal Corporations.

Thus, the spirit of commons is at the core of our initiative, we are happy to seek more support from various diverse communities. Budget researchers, policy makers, civil societies, journalists and data contributors can reach out to us at info@openbudgetsindia.org. While technologists, data scientists, visualizations experts and designers can directly collaborate with us in our design and development cycle via Github. Together we aim to continue our efforts in making India’s budgets open, usable and easy to comprehend.