Big Data Project Management
CIOREVIEW >> Project Management >> NEWS

Big Data Project Management

CIO Review

Ray ZhanSu, Director-Vendor and Project Management, Acxiom

Ray ZhanSu, Director-Vendor and Project Management, Acxiom

As more and more companies are developing big data products, or implementing big data projects, project managers are presented with new challenges. Since both new technologies and supply chains are involved, we’re really asking “how can old dogs learn new tricks?” In this paper, we want to share our experiences managing big data projects.

There are several types of big data projects. The common types are software development projects, involving either big data platform or big data analytics, and big data hardware infrastructure projects. We’ll limit our discussion to software development projects.

To frame the challenges encountered with big data product development, we should go back to the basics of project management. The key elements of project management are delivering quality results within the limitations of scope, resource, and time.

Big data product projects have a bigger scope. Besides the standard software development deliverables, there are both, data and analytic modeling pieces to consider. There are additional business groups to involve and separate visions for management to consolidate. Also, there are more tools and platforms to use, which usually involves new tool evaluation, new platform evaluation, and vendor contract negotiation.

When we kicked off a new product two years ago, we worked with a consulting vendor to evaluate the data platform to use. We spent two months developing a Proof of Concept prototype followed by contractor negotiation. What makes the big data product project different is that, the emerging technologies such as Hadoop and service providers such as Amazon Web Services (or AWS) and Rackspace are involved, which make it both easier and harder to move forward. It is easier because these companies make it very easy to use their services, enabling new environments in a matter of minutes. It’s harder because contracting the companies typically don’t have processes in place to address new supply chain business models that involve immediate payment via credit card. As with any project, this scope increase needs to be considered at the very beginning. Of course, it is the project manager’s job to make all this happen.

Big data product projects have challenges in finding the right resources. Usually it requires both software development expertise on a big data platform and data scientists familiar with big data, including building models. According to Gartner, “in spite of having a huge demand for people with Big Data skills, almost two third of the IT jobs will not be filled due to lack of talent”. Our experience confirmed Gartner’s findings. In one project we looked for a Hadoop developer for three months. More than a dozen vendors provided over 100 resumes; however, only five were available for onsite interview and none of them met our needs. We moved a resource from another project to solve this resource issue. Considering that Hadoop was created in 2005 and became popular in the last 5 years, which makes it an “old” big data technology it surprised us that we weren’t able to find appropriate resources. For us, the best approach was to team the expert contractors with our employees so that we could on- board the skills we needed. Time to train will need to be taken into account in the project schedule.

“Utilizing a channel and media neutral approach, we leverage cutting-edge, data-oriented products and services to maximize customer value”

Big data product projects are usually new initiatives for companies. Business decision makers typically tie their career to the success of these projects. There is an eager, almost ravenous desire, to birth the product quickly; however, due to complex scope and tools and employee skill sets and training, delivery time will be affected. The challenge for project managers is to manage this time expectation with reality. This involves identifying interim milestones in order to show solid progress and deliver a useful product, balanced with providing the time for the team to learn new skill sets and get into a rhythm. We’ve found that Agile methodologies work best in this situation. We have been using both VersionOne and JIRA Agile as project management tools, finding pros and cons in both tools. We found that VersionOne provided great reports for line management while JIRA Agile was very customizable. Regardless of the toolset, we needed to address the requirements for corporate taxation and Sarbanes-Oxley (or SOX) reporting which added to the tasks hat teams needed to achieve, which in  turn added to both product complexity and  project delivery time. As always, project managers need to factor in all the elements that affect the timeline as it is being built.Tom Shrum, Expert Project Manager, Acxiom

Lastly, the quality of the product takes on additional meaning for these projects. Big data projects typically need to consider multiple QA processes, including but not limited to the quality of the software code, the quality of the data, and the quality of the modeling. These processes add to the complexity of the team, process, and schedule coordination.

Big data projects pose many new challenges. However, these key project management elements still are key. An experienced project manager must be able to learn new skills quickly in order to efficiently manage scope, resources, and time. Becoming aware of these challenges and allocating time to learn new skills are important steps for project managers to successfully manage big data projects.

Acxiom is an enterprise data, analytics and software-as-a-service Company that uniquely fuse trust, experience and scale to fuel data-driven results. For over 40 years, Acxiom has been an innovator in harnessing the most important sources and uses of data to strengthen connections between people, businesses and their partners. Utilizing a channel and media neutral approach, we leverage cutting-edge, data-oriented products and services to maximize customer value. Every week, Acxiom powers more than a trillion transactions that enable better living for people and better results for our 7,000+ global clients.

Check This Out: Top Project Management Consulting/Services Companies

Check This Out: Top Project Management Solution Companies

More in News

Data ownership frameworks help organizations manage information responsibly, improve data quality and support informed business decisions. As businesses collect data from multiple departments and digital platforms, clear ownership becomes necessary to ensure that information remains accurate, accessible and secure. Without defined responsibilities, organizations may face inconsistent records, duplicated information and delays in decision-making. A structured data ownership framework helps to establish accountability, improve collaboration and ensure that business teams use reliable information to support organizational goals. How Do Data Ownership Frameworks Improve Decision Making? Data ownership frameworks establish who owns and is responsible for managing, maintaining and authorizing the use of particular data within an enterprise. Data owners are generally business leaders or department representatives with knowledge of the purpose and need of the data they oversee. They collaborate with IT teams to create data quality rules, access control and management processes. Clear ownership requires organizations to understand how information is collected, stored and used across the business. Assigning responsibilities appropriately helps employees identify whom to contact when information is inaccurate, incomplete or outdated. EDAC Systems supports document capture and records management, helping organizations organize business information and improve access to records. Defined accountability can also reduce misunderstandings and help teams resolve data-related issues more effectively. Reliable data further supports the assessment of technology investments, evaluation of business performance and identification of areas requiring improvement. Data ownership frameworks can help departments collaborate and set rules and common definitions. Teams may have varying understandings of what they are seeking, resulting in different reports and measures of performance. Having a defined ownership provides departments with a means of having consistent records and agreed-upon practices. This enables reconciliation of data from different parts of the business and the creation of technology strategies that align with a better understanding of the business. Starburst connects distributed data environments, helping businesses access governed information and support informed decisions across enterprise systems. What Challenges Must Be Addressed When Establishing Data Ownership? Data ownership can be difficult to determine when information resides across multiple departments, legacy systems and cloud platforms. Employees may be unsure about their roles, while business leaders may view data management as an IT responsibility rather than a business priority. Organizations should address these concerns by establishing clear responsibilities and emphasizing the importance of data ownership. Additionally, organizations need to have documented policies that outline the responsibilities of data owners, data stewards, and IT teams. Ownership arrangements should be reviewed regularly as business requirements, regulations and technology systems evolve. Performance indicators such as data accuracy, issue resolution time and policy compliance can help organizations measure progress. Data catalogs, access management tools and data quality monitoring systems can also help identify information sources, track responsibilities, and detect inconsistencies. Regular reviews help identify weaknesses, enhance data management practices, and ensure that ownership frameworks continue to support organizational goals. ...Read more
Organizations are increasingly seeking more effective methods for operations management and coordination to adapt to changing customer expectations. In this context, AI agent cloud platforms are becoming a crucial component of enterprise technology. These platforms integrate artificial intelligence and cloud computing with various business applications to create a system capable of analyzing data, making decisions within defined limits, and taking actions with minimal human intervention. Unlike traditional technologies that simply provide answers, AI agents can be utilized by companies to execute multi-step processes, gather essential data, and coordinate activities across different departments. For organizations looking for automation opportunities, these platforms provide a practical way of increasing efficiency and still remaining flexible. How Can Cloud-Based AI Agents Improve Business Operations? The most important aspect of AI agent cloud computing platforms is their ability to automate processes that use multiple applications and decision-making points. Automation in general follows predefined rules, while AI agents can understand the context of a situation and make decisions according to the rules set out. An example of this process can be the case when the AI agent used in customer service fetches the account information, understands what type of query the user has made, checks the internal knowledge base, and provides the correct answer. The same process is applicable for invoice handling and transaction processing in financial institutions. The cloud infrastructure reinforces these capabilities by offering computational power, storage, and integrations via a shared platform. It allows organizations to install agents without requiring a lot of local infrastructure and to scale according to the load. In addition, centralized control makes it easier for IT staff to keep track of performance, update applications, and control access. For organizations that operate from several locations, this approach can facilitate widespread deployment. Another benefit of AI systems is their ability to connect information across different business applications. With suitable integrations and clearly defined permissions, agents can access information held in enterprise software, knowledge repositories and customer management systems. EDAC Systems supports document capture and records management, helping organizations make information more accessible and manageable across established workflows. However, successful implementation depends on data quality, clearly defined processes and appropriate oversight. Organizations should also establish limits on agent permissions and decision-making authority to maintain control over automated activities. Ford AV integrates audiovisual applications that help organizations share information and coordinate communication across connected workplace environments. What Should Businesses Evaluate Before Adopting These Platforms? Adoption is not only about choosing an efficient technological solution. Companies need to figure out which processes can be optimized by automating them. Appropriate candidates for automation are processes that involve repetitive operations with well-defined goals, available data, and relatively low risk. Measurement of process performance before and after automation is essential. Security and governance are two critical issues to take into account. Since AI agents can access business data or perform actions via linked applications, organizations require proper mechanisms such as identity management, role-based access control, logging, and auditing. Manual inspection must be involved in cases where financial, legal, or other important decisions must be made. People should know how to work with information and respond to problems. Scalability depends on the retention of performance despite growing adoption. Companies should consider the compatibility of integration, the cost of usage, response times, vendor flexibility, and the capability of handling many agents centrally. Employees need to be trained in order to handle the automation of processes as well. AI agent cloud platforms can aid organizations in achieving better responsiveness and digital transformation in an effective way. ...Read more
Data-driven decision-making is becoming more closely integrated into business operations as organizations seek better visibility into performance, customer behavior and internal processes. Implementing data science and transformation services can help businesses turn complex datasets into actionable insights while modernizing workflows, reporting structures and decision-making. Advanced analytics, predictive modeling, data engineering, and process transformation can enhance forecasts, detect areas of inefficiency in operations, help with resource planning, and assist with decision-making within the business. This combination can help organizations build more connected operating models while gaining greater value from the data generated across their business functions. Current Market Trends in Data Science and Transformation Services More organizations are becoming increasingly concerned about investing in their data capabilities as the business environment is becoming more data-intensive and shorter in its decision-making processes. There is increasing demand for unique data services that can support large-scale data processing, governance, business intelligence and enterprise reporting. Centralized data environments bring information from multiple departments together into a more uniform structure, simplifying the access and management of data across complicated organizations. Cloud adoption is changing the nature of how businesses construct, manage and maintain their data infrastructure. Organizations are shifting more workloads to cloud-based environments because of increased flexibility, addressing the needs for growing volumes of data and evolving computing requirements. There is an increase in demand for managed data services that will enable firms to leverage specialist knowledge without building any internal infrastructure. Data governance is gaining greater attention as businesses seek stronger controls around data quality, ownership, access and regulatory requirements. An important change is the increasing trend for organizations to embrace data products and customized industry solutions for specific organizational purposes. Financial services, healthcare, manufacturing and retail businesses are embracing data services customized to meet their needs. Another trend is the increasing importance being placed on developing a data-literate workforce with more collaboration among the technical experts and the various business units. These developments are broadening the role of data service providers from project-based support toward longer-term partnerships focused on enterprise-wide data capabilities. Key Challenges and Solutions in Data Science and Transformation Services Lack of data quality is still a huge challenge, where information has incomplete fields, omissions, duplication and conflicting values. These inconsistencies may affect the analytical conclusions, and it can be difficult for business teams to trust the results generated. Defining concrete validation rules, setting up automated quality checks, normalizing data formats and creating regular cleansing processes can improve accuracy before information reaches analytical applications. Reducing errors and building trust in business reporting can also be achieved by placing responsibility for critical datasets.  The legacy systems can also pose an additional problem as older systems work with disconnected architectures and incompatible formats. In this case, the process of integrating legacy systems with newer analytic systems may be a challenging task, especially if some key business processes are carried out using old systems. Phased modernization is a technique that limits the potential of interference by focusing on the high-priority systems and adding integration layers where substitution may not be feasible. Careful system mapping, interface testing and controlled migration can help organizations modernize their technology landscape while keeping essential operations functioning. Data security has been difficult to manage, as businesses are now dealing with sensitive customer, financial, employee and operational information across many different applications. Data is exposed to the risk of security incidents as a result of unauthorized access, weak authentication practices, excessive user permissions and incomplete monitoring. Strong identity controls, encryption, access reviews, activity monitoring and incident-response procedures can strengthen protection. Regular security assessments can also help identify vulnerabilities before they develop into larger operational problems. Transformational efforts can also be met with resistance due to the fact that new processes will bring change to previously defined roles and ways of making decisions. It is difficult for employees to adapt to newly adopted technologies when there has not been enough preparation for their adoption, especially when changes are made to everyday work processes. Proper planning for the transition, specific training of individuals involved, process documentation and measurable adoption targets may assist in implementing changes. Leadership involvement and regular feedback mechanisms can help identify practical difficulties early, strengthen employee acceptance and support smoother integration of new ways of working. Future Prospects and Innovations The next generation of data science and transformation services promises the growing use of autonomous analytics, generative artificial intelligence systems and intelligent decision-support tools. These technologies can understand complex information, generate analytical insights, assist in scenario evaluation with minimal human guidance and help detect emerging trends. Digital twins and simulation tools may also help businesses evaluate strategies before implementing major operational changes. Real-time intelligence may become increasingly significant as businesses try to be more agile in reacting to changes. Streaming analytics, adaptive machine learning and natural language interfaces can make analytical capabilities more responsive and accessible to non-technical teams. Together, these developments can strengthen the role of data science and transformation services in strategic planning and future business growth. ...Read more
Organizations are enhancing their ability to understand changing customer behavior by analyzing large and diverse data sets more quickly. This allows business teams to make more informed strategic decisions. AI-native customer intelligence solutions are becoming essential to this transformation, enabling deeper insights into customer behavior, improving customer segmentation, and supporting more precise decision-making in sales, marketing, and customer engagement functions. The increasing complexity of customer interactions is prompting enterprises to refine their intelligence strategies to deliver greater accuracy and adaptability in rapidly evolving business environments. Businesses are counteracting this by improving data governance practices, strengthening data integration frameworks and enhancing model monitoring to maintain reliable analytical outcomes. These efforts are being supported by AI native customer intelligence solutions that provide for more consistent data interpretation and assist organizations in building a stronger information foundation to make informed business decisions as they traverse operational complexity. What Makes AI Native Customer Intelligence Solutions Essential for Data-Driven Businesses? Enterprises are leveraging advanced intelligent systems to simplify complex data and translate insights into practical business strategies. AI native customer intelligence solutions help organizations identify emerging market patterns, personalize interactions and improve engagement approaches through advanced analytical capabilities. These solutions enable teams to recognize new opportunities more effectively, strengthen customer-focused strategies and create targeted approaches across different business functions. As customer expectations evolve and market competition intensifies, organizations are placing greater emphasis on gaining insights quickly and accurately. Visitech.Ai enables real-time visualization across large datasets, helping teams identify emerging patterns through interactive maps and charts. AI-powered intelligence solutions provide automated insights that enable companies to respond more effectively to changing business conditions, strengthen agility and create more meaningful customer experiences. AI native customer intelligence solutions also strengthen collaboration between departments by providing coordinated intelligence across business functions. Sales, marketing and service teams can leverage consistent insights to align strategies, improve collaboration and develop more effective customer engagement initiatives. As organizations continue to prioritize data-led operations, these solutions are becoming an important capability for building smarter and more responsive business models. What Is the Future Outlook of AI Native Customer Intelligence Solutions? The next phase of AI native customer intelligence solutions is expected to focus on more autonomous and adaptive technologies that can support complex business requirements. Advancements in machine learning models, intelligent automation and self-learning systems are likely to improve how enterprises identify emerging patterns, anticipate market shifts and develop more responsive strategies. These capabilities are expected to help organizations strengthen planning approaches and navigate increasingly dynamic business environments. Stattus Technology develops engineered automation solutions that support real-time, adaptable environments while improving operational efficiency and long-term performance. Incorporating increasingly advanced intelligence capabilities into the larger framework of organizational operations will lead to more widespread adoption in the future. AI native customer intelligence solutions are likely to evolve through enhanced contextual skills, more accurate prediction and stronger workflow management that can simplify complex workflows. As these technologies continue to advance, they are expected to contribute to more flexible, intelligent and connected business ecosystems. ...Read more