code4craft/webmagic
Fork: 4172 Star: 11252 (更新于 2024-05-07 14:01:01)
license: Apache-2.0
Language: Java .
A scalable web crawler framework for Java.
最后发布版本: WebMagic-0.10.0 ( 2023-12-05 12:51:35)
A scalable crawler framework. It covers the whole lifecycle of crawler: downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler.
Features:
- Simple core with high flexibility.
- Simple API for html extracting.
- Annotation with POJO to customize a crawler, no configuration.
- Multi-thread and Distribution support.
- Easy to be integrated.
Install:
Add dependencies to your pom.xml:
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-core</artifactId>
<version>${webmagic.version}</version>
</dependency>
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-extension</artifactId>
<version>${webmagic.version}</version>
</dependency>
WebMagic use slf4j with slf4j-log4j12 implementation. If you customized your slf4j implementation, please exclude slf4j-log4j12.
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
Get Started:
First crawler:
Write a class implements PageProcessor. For example, I wrote a crawler of github repository infomation.
public class GithubRepoPageProcessor implements PageProcessor {
private Site site = Site.me().setRetryTimes(3).setSleepTime(1000);
@Override
public void process(Page page) {
page.addTargetRequests(page.getHtml().links().regex("(https://github\\.com/\\w+/\\w+)").all());
page.putField("author", page.getUrl().regex("https://github\\.com/(\\w+)/.*").toString());
page.putField("name", page.getHtml().xpath("//h1[@class='public']/strong/a/text()").toString());
if (page.getResultItems().get("name")==null){
//skip this page
page.setSkip(true);
}
page.putField("readme", page.getHtml().xpath("//div[@id='readme']/tidyText()"));
}
@Override
public Site getSite() {
return site;
}
public static void main(String[] args) {
Spider.create(new GithubRepoPageProcessor()).addUrl("https://github.com/code4craft").thread(5).run();
}
}
-
page.addTargetRequests(links)
Add urls for crawling.
You can also use annotation way:
@TargetUrl("https://github.com/\\w+/\\w+")
@HelpUrl("https://github.com/\\w+")
public class GithubRepo {
@ExtractBy(value = "//h1[@class='public']/strong/a/text()", notNull = true)
private String name;
@ExtractByUrl("https://github\\.com/(\\w+)/.*")
private String author;
@ExtractBy("//div[@id='readme']/tidyText()")
private String readme;
public static void main(String[] args) {
OOSpider.create(Site.me().setSleepTime(1000)
, new ConsolePageModelPipeline(), GithubRepo.class)
.addUrl("https://github.com/code4craft").thread(5).run();
}
}
Docs and samples:
Documents: http://webmagic.io/docs/
The architecture of webmagic (refered to Scrapy)
There are more examples in webmagic-samples
package.
License:
Licensed under Apache 2.0 license
Thanks:
To write webmagic, I refered to the projects below :
-
Scrapy
A crawler framework in Python.
-
Spiderman
Another crawler framework in Java.
Mail-list:
https://groups.google.com/forum/#!forum/webmagic-java
http://list.qq.com/cgi-bin/qf_invite?id=023a01f505246785f77c5a5a9aff4e57ab20fcdde871e988
QQ Group: 373225642 542327088
Related Project
-
A web console based on WebMagic for Spider configuration and management.
最近版本更新:(数据更新于 2024-05-09 03:27:05)
2023-12-05 12:51:35 WebMagic-0.10.0
2023-09-10 17:19:22 WebMagic-0.9.1
2023-06-22 11:25:37 WebMagic-0.9.0
2022-11-24 00:52:02 WebMagic-0.8.0
2022-10-24 22:59:46 WebMagic-0.7.6
2022-09-02 19:35:35 WebMagic-0.7.5
2017-07-30 15:43:36 WebMagic-0.7.3
2017-06-17 16:15:18 WebMagic-0.7.2
2017-06-04 18:28:23 WebMagic-0.7.1
2017-05-29 14:14:34 WebMagic-0.7.0
主题(topics):
crawler, framework, java, scraping
code4craft/webmagic同语言 Java最近更新仓库
2024-05-08 08:13:42 SeleniumHQ/selenium
2024-05-07 19:46:08 dromara/dante-cloud
2024-05-07 07:23:41 Azure/azure-sdk-for-java
2024-05-05 02:37:11 FAForever/downlords-faf-client
2024-05-03 19:50:08 spring-gradle-plugins/dependency-management-plugin
2024-05-03 18:43:13 elastic/elasticsearch