Danit Shifman Abukasis

This research presents a novel two-phase approach aimed at optimizing website crawling for a financial software company, addressing the challenge of combining resource constraints and classification costs. The first phase utilizes a machine learning model to predict the probability of an efficient crawl. In the second phase, the probabilities of the previous phase are integrated into an optimization model. This model is designed to allocate crawling resources in a way that maximizes efficiency while taking into account both global and local constraints. The global constraint refers to a limitation on the total number of crawls, and the local constraints are the limitations to specific dataset groups, such as a limitation to every website provider. We present a mathematical formulation and solution of the problem, leveraging a cost matrix and constraint equations to minimize classification costs while ensuring compliance with global and local restrictions, and prove that the problem has an integer solution. An experimental study demonstrates the effectiveness of our approach in optimizing daily crawls. The proposed method offers a scalable solution for handling the problem of web crawling in resource-constrained scenarios, providing a significant contribution to the fields of machine learning and operations research.