Big data in smart farming

C6:
Dr Wida Susanty haji suhaili
Smart Farming
• Smart farming is a farming management concept that uses information and
communication technology (ICT) to increase the quantity and quality of
agricultural products.
• Precision agriculture is about implementing automatically controlled
machines, monitoring of the yields and different ways of fertilizer spreading
and seed drilling.
• With smart machines and sensors cropping up on farms, there is an
abundance of data both in quantity and scope. The entire farming is now
data-driven and data-enabled.
• Technologies in smart farming include tracking systems like Global Positioning
System (GPS) and others such as IoT, big data, analytics and cloud computing,
How big data can help agriculture
• IoT devices help in data collection. Sensors plugged in tractors and trucks, in the fields,
soil, and plants aid in the collection of real-time data directly from the ground.
• Integration of large amount of data collected with other information available in the
cloud, such as weather data and pricing modes to determine patterns
• Patterns and insights assist in controlling the problem. Help to pinpoint existing issues,
like operational inefficiencies and problem with soil quality and formulate predictive
algorithms that can alert even before problem occurs.
• Digital Transmission Network – provides agricultural information solution and market
intelligence to customers.
• Help farmers and commodity to access up-to-date weather and pricing data to better
manage their business
• Historical weather data, satellite, drone images and soil types – make informed decisions
Food processing stages
• Aim: Less food waste, more profits for farmers
• Boost food processing. After the harvest, the produce undergoes
several stages of handling:
• Processing (which includes cleaning up fruits and removing damaged parts),
• Grading (each plant is assessed according to its look and general “ugliness”, as in
some parts of the world it is common for the law to rule that only the highest quality
products can make it to the store),
• Packaging (making sure that the material is protective enough is vital to ensuring the
quality of product at all times during the transportation process),
• Storage and ripening (adequate temperatures and storing conditions are needed to
ensure the proper maintenance of harvested fruits or vegetables).
Food processing stages
• Processes automated with the help of technologies based on Artificial
Intelligence and Internet of Things help in making sure that the
produce is being handled correctly at each stage, and is not
mismanaged.
• Taking advantage of sensors and devices that monitor the storage
conditions helps in reducing the waste of food.
• Adequate temperature, humidity, levels of oxygen, carbon dioxide,
and ethylene need to be ensured at all times, so as many products can
leave the farm in the best state possible, ready for sale. 
Smart Farming Technologies
• GPS enables data to be allocated to a particular area of the land or help to determine the
present location of machines or animals in the barn.
• Sensors in the farm help to measure mineral content, such as the nitrogen content or
abundance of weed.
• Data collected from these sensors help to calculate the best fertilizer composition for that
particular area and subsequently administer fertilizer as per the set rules and regulations.
• Farmers are also using drones to control the field and plant growth.
• Images captured by these drones are used to collect information aout the entire farm area,
create a detailed digital map for better administration
• For livestock farming, microchips and sensors measure the movement pattern of animals, body
temperature and other vital data.
• For cows, analyzing data not only monitors the health of cows but also help to determine the
time of insemination.
Uses of big data in smart Farming
• Yield production
• Uses mathematical models to gain insight around the weather, yield, chemicals, leaf, and biomass index
along with machine learning which crunch statistics and enhance power making decisions.
• Predicting yields this way allows the farmer to have an insight into the kind of plant and its location to be
planted.
• Risk management
• Farmers can leverage on big data to evaluate the chances of uncertain events like crop failure, drought and
the shift in weather patterns and other calamities.
• Feed efficiency within the production of livestock is enabled by leveraging big data.
• Food safe and prevention
• Collection of data which centers on temperature, humidity, chemical proportionpains a complete picture of
the health of smart agricultural business.
• Earlier detection helps to lower cost and also reduce wastage.
• Operational Management
• Aid in everyday maintenance of agricultural land
• Sensors in the vehicles provide a lot of data which can assist in fleet and equipment management, leading
to reduced downtime and increased productivity.
 Top 4 use cases for big data on the farm
• Feeding a growing population (improve farm yield)
• Big data provides farmers granular data on rainfall patterns, water cycles, fertilizer requirement ns and more.
• Enable to make smart decisions, what crops to plant for better profitability and when to harvest.
• Using pesticides ethically
• Big data allows farmers to manage administration of pesticides by recommending what pesticides to apply,
when, and by how much
• Farmers can adhere to government regulations and avoid overuse of chemicals in food production.
• Result in increase profit since crops don’t get destroyed by weeds and insects
• Optimizing farm equipment
• Large farm, integrated sensors in the farming equipment and deployed big data application
• This can help monitor tractor availability, service due dates and fuel refill alerts.
• Optimize usage and ensure the long-term health of farm equipment
• Managing supply chain issue
• A third of food produced for human consumption is lost or wasted yearly.
• To address this, food delivery cycles from producer to the market need to be reduced.
• Big data can help achieve supply chain efficiencies by tracking and optimizing delivery truck routes
How is big data analytics used in farming
• Digital transformation in agriculture introduces new technologies such as aggrotech which
are helping farmers with making decisions based on data rather than their intuition and
claims that everyone dies it the same way.
• Software that helps in monitoring the state of the land, livestock and equipment.
• GPS navigation combined with AI and the Internet of Things enables planning agricultural
work in a more efficient and smoother way, easily adjusting to various weather conditions
and other factors
• Irrigating the land, eliminating pests, applying fertilizers, detecting illnesses among the
livestock, and other potential issues even before they occur. the existing tools can provide
them with extra leverage that will help them care about their land, optimize the use of
resources and maximize the profits.
• A properly set up infrastructure can also help in managing the crops after they are picked
up – monitoring the storage conditions, i.e. proper temperature or humidity level. 
Challenges and use cases
• Big data encompasses data from various digital devices, software and applications.
• Making it possible to process a large amount of information from various sources.
• Big data ensures the interoperability of digital tools to organize, analyze and
exploit data.
• This comes in the form of integrated consoles in various machines:
• Weather stations, tractors, combine harvesters etc.
• The computerized devices connected to the agricultural machines detect a lot of
information:
• Areas covered, interventions carried out, quantity of products, harvested and climatic
indications.
• Decision support tools (DST) interventions are easier and more relevant.
• Precision agriculture pilots the crops to better store observation, visualize the interventions
made and use inputs.
• DMOs allow technical operators to monitor crops remotely and optimize field visits.
Data Chain of Big Data Applications (Chen
et al (2014)
Farm processes
• Data collected from the field or the farm include information on
planting, spraying, materials, yields, in-season imagery, soil types,
weather and other practices.
• There are 3 categories of data generation.
• Process-mediated (PM)
• Machine-generated (MG) and
• Human-sourced (HS)
Farm Processes
• Process-mediated (PM)
• PM data, or the traditional business data, result from agricultural processes that record and
monitor business events of interest, such as purchasing inputs, feeding, seeding, applying
fertilizer, taking an order, etc.
• PM data are usually highly structured and include transactions, reference tables and
relationships, as well as the metadata that define their context.
• Traditional business data are the vast majority of what IT managed and processed, in both
operational and business information systems, usually structured and stored in relational
database systems.
• Machine-generated (MG)
• MG data are derived from the vast increasing number of sensors and smart machines used to
measure and record farming processes; this development is currently boosted by what is
called the Internet of Things (IoT).
• MG data range from simple sensor records to complex computer logs and are typically well-
structured. As sensors proliferate and data volumes grow, it is becoming an increasingly
important component of the farming information stored and processed.
• Its well-structured nature is suitable for computer processing, but its size and speed is beyond
traditional approaches.
 Farm Processes
• Machine-generated such as:
• For Smart Farming, the potential of unmanned aerial vehicles (UAVs) has been well-recognized.
• Drones with infrared cameras, GPS technology, are transforming agriculture with their support for better decision
making, risk management .
• In livestock farming, smart dairy farms are replacing labour with robots in activities like feeding cows, cleaning the
barn, and milking the cows.
• On arable farms, precision technology is increasingly used for managing information about each plant in the field.
• With these new technologies data is not in traditional tables only, but can also appear in other formats like sounds or
images.
• In the meantime several advanced data analysis techniques have been developed that trigger the use of data in images
or other formats
• Human-sourced (HM)
• HM data is the record of human experiences, previously recorded in books and works of art, and later in
photographs, audio and video.
• Human-sourced information is now almost entirely digitized and stored everywhere from personal computers to
social networks.
• HM data are usually loosely structured and often ungoverned. In the context of Big Data and Smart Farming,
human-sourced data have rarely been discussed except in relation to the marketing aspects.
• Limited capacity with regard to the collection of relevant social media data and semantic integration of these data
from a diversity of sources is considered to be a major challenge
 Big Data Applications

• Table provides an overview of current Big Data applications in relation to different

elements of Smart Farming in key farming sectors.
• From the business perspective, the main data products along the Big Data value chain are
(predictive) analytics that provide decision support to business processes at various levels.
• The use of analysis of sensor data or similar data must somehow fit into existing or
reinvented business processes, integration of data from a variety of sources, both
traditional and new.
Key areas of change
• Real-time forecasting
• Tracking of physical items
• Reinventing Business process

• Thus this will likely change both farm structures and the wider food
chain in unexplored ways as what happened with the wider adoption
of tractor and the introduction of pesticides.
• Big data applications are driven by push-pull mechanisms.
• Pull – there is a need for new technology to achieve certain goals
• Push – new technology enables people or organizations to achieve
higher or new goals.
Value-added services with data extraction
• Big data deals with the last stage of smart farming procedures, the
information extracted by various relevant analytic techniques is used
in value-added services that are provided to users.
• Users can be farmers, researchers, food companies, or the
government.
• The value-added services can be utilized by decision support systems
to improve farming practices such as crop health monitoring, yield
prediction, water management, demand forecasting, pesticide and
fertilizer management, animal behavior monitoring, livestock health
and welfare monitoring, and livestock feed consumption monitoring.
Data security and data privacy
• Data security protects data from adversary attacks,
• while data privacy governs how data is collected, analyzed, stored and
accessed.
• Data security is more about protecting data from attacks and
• data privacy is more focused on using data responsibly according to
the users’ desire, and protecting data from unauthorized access.
Big Data Privacy • This lifecycle provides a better intuition about privacy threats,
Lifecycle in requirements and their correlation in different stages of smart
farming.
Smart Farming • Data collection – raw data collected from devices (sensors).
Sensors deployed are limited in energy and computation
resources. Only capable of performing very simple data
refinement and processing tasks.
• Data transfer – collected data transferred to the servers.
Combination of different technologies is utilized including Wi-Fi,
cellular, LAN and Bluetooth.
• Data storage – Stored locally or cloud storage server. Data
ensured with effective access control mechanisms to prevent
unauthorized data inquiries.
• Data analytics – Data are analyzed and exploited to extract
knowledge through the use of analytics methods.
• Data-driven services – Services that can provide a wide range of
recommendations to farmers. To improve profitability of their
business using –data-driven services.
• Data destruction – Data no longer useful should be erased based
on the preliminary agreements.
 Privacy-Preserving Technologies in Smart Farming
• Machine Learning
• Machine learning is a method to enable machines to learn based on their experience.
• Using machine learning, the farmers can analyze the collected data and discover patterns, predict the future, and increase the performance
of their businesses.
• In smart farming, the deployed sensors are responsible to collect data from different aspects of the farm, and then this data will be
analyzed by machine learning models to provide an insightful analysis regarding diverse applications, such as yield prediction, quality
assessment, water management, disease prediction, and livestock monitoring.
• Because machine learning models utilize data for the learning process, it is essential to design the learning models in a way that sensitive
information remains protected [74].
• Edge Computing
• There are two main types of communication among devices in many smart farming structures. One is device-to-device communication, in
which the digital devices communicate directly, with no intermediate. The second class of communication is device-to-server, in which IoT
devices make independent connections to data and computation servers (usually cloud servers) to transfer data.
• Another communication class where digital devices use middleware, called the edge nodes, to transfer data through the network. This
middleware collects data from end-users (e.g., sensors) and performs some pre-processing tasks, such as data or protocol change
algorithms to reduce the volume of transferred data.
• Edge nodes usually have more computational power than the sensors and can process larger tasks on data before transforming through the
network.
• This enhances data privacy by reducing the volume of transmitted data in the network. By performing pre-processing tasks at the edge of
the network, it is not necessary to transfer all the raw data to the cloud servers, and most of these data will stay in private storage, which
enhances data protection.
• The other advantage of making use of edge-enhanced architecture is having more computing resources, which enables using more
sophisticated algorithms for data aggregate encryption before transmission through the network
Privacy-Preserving Technologies in Smart Farming
• Blockchain
• This technology is a distributed ledger that records all previous transactions on the public
ledgers and utilizes mathematical algorithms to prevent data manipulations and certify the
validation of data.
• Blockchain reduces privacy concerns by eliminating the central point of vulnerability in the
system. All clients can have a copy of the ledgers, and nobody has full control to store, use,
and delete the whole data. Data on the blockchain is transparent and immutable; therefore,
all previous records are traceable.
• Blockchain platforms also deploy a cryptographic private key, in addition to the ring
signature, ensuring privacy and confidentiality for the users [96].
• Different blockchain-based mechanisms have been introduced in recent years. Some
technical features that can describe the difference among these architectures are:
• (1) The permission mechanism which indicates the method that is utilized for authentication.
• (2) The consensus algorithm that is the mathematical algorithm to decide how to add a new
block to the blockchain.
• (3) The smart contract/cryptocurrency which points out if the blockchain is used for smart
contract deployment, financial payment, or both
Privacy Concerns on Data Sharing Practices
• Ownership
• Collection
• Notice
• Choice
• Portability
• Disclosure
• Data retention and availability
1.Ownership: Farmers own the information collected in the farming operations, but it is also their
responsibility to negotiate and agree on data sharing with other parties. It is also the farmers’
responsibility to make sure only the data they own have been included in an agreement.
2. Collection, Access, and Control: Any access and use of farm data should be explicitly permitted by
the farmer in the signed contracts or other types of legal agreements.
3. Notice: Not only should farmers be clearly informed that their data are being collected, but they
must also be aware of how their data are going to be used and disclosed by the technology provider or
any other third party. These notices should be provided in an easily located and readily accessible
format.
4. Choice: Technology providers must provide choices for farmers to opt in, opt out, or cancel the
services.
5. Portability: The farmers should be able to transfer their data to other systems, except for the data
that have been aggregated or anonymized.
6. Disclosure, Use, and Sale Limitation: The technology provider should notify the farmer if they decide
to sell or disclose the collected data, and they should also provide the chance for the farmer to cancel
the service or remove the data. Working with a new third party should be based on the agreements
consistent with the primary agreement between the parties.
7. Data Retention and Availability: Technology and service providers should provide mechanisms to
erase or return farm data based on the farmer’s request, either immediately or after an agreed-upon
timespan.
 Challenges and Future Research Directions
• Standardization:
• Numerous IoT devices are connected through the IoT. These devices utilize various technologies, configurations, and
protocols. The heterogeneity of these devices is an issue in developing privacy-preserving solutions to work in different
farming practices.
• Without standardization, the interconnection mechanisms in agriculture are highly complex because different
technology providers use different formats for their system operations such as sensing, transmission, storage, routing,
and service management.
• Standardization addresses this issue by providing unified approaches to be followed by technology providers and
farming actors, consequently improving the privacy-preserving solutions for smart farming.
• Trust:
• Previous studies show that trust is a bottleneck in data sharing and collaborative service production in smart farming
[6].
• Trust evaluation is a complimentary act to the data protection mechanisms provided in farming applications. Trust
evaluation reduces the concerns of agricultural sectors regarding data sharing procedures and encourages these actors
to participate in collaborative value-adding services.
• To this end, the trustworthiness of an actor in the agricultural ecosystem can be evaluated based on investigating the
background of the actor and analyzing recommendations from other businesses that have previous experience in
working with the actor.
• Automatic tools and services that manage this process accelerate the trust evaluation process and reduce the threats to
the privacy of data in smart farming.
 Challenges and Future Research Directions
• Legal Frameworks:
• Smart farming requires legal frameworks that clearly discuss different aspects of data privacy, including responsibilities and
accountability.
• These frameworks can indicate the requirements for different agricultural sectors to access, utilize, and make profits from
available data.
• Such best practices should also determine the responsibilities of farming actors in each stage of the data lifecycle and set
requirements to ensure data privacy.
• Moreover, the legal frameworks should clearly elaborate on the accountability of each actor and the potential
consequences in the case of data leakage.

• Blockchain:
• Blockchain is a promising technology that has gained considerable attention in smart farming.
• Despite the advantages of this technology, some concerns and issues have affected the development of blockchain-based
solutions in real-world agricultural applications.
• A major concern in this regard is scalability. Since numerous smart devices are working in farming applications, the
blockchain-based method must be able to provide solutions that manage all the transactions transmitted through the
system in a reasonable time.
• The other concern in leveraging the blockchain in farming is that the mining procedures commonly require a great amount
of computation and energy resources, which is a constraint in many farming smart devices.
References
• Wolfert, S. et al (2017) Big Data in Smart Farming – A review,
Agricultural Systems, Elsevier
• Toshendra S (2019), Role of Big Data in Smart Farming,
https://www.globaltechcouncil.org/big-data/role-of-big-data-in-smart
-farming/
• Big data and smart farming: the importance of data to optimize
agricultural practices.
https://www.synox.io/en/smart-agriculture/smart-farming-big-data/
• Big data and agriculture: A complete guide
https://www.talend.com/resources/big-data-agriculture/