twitter-scraper - Effortlessly Retrieve Twitter Data with No API Constraints

Twitter Scraper: A Comprehensive Guide

The Twitter Scraper project is a robust library designed to simplify the process of retrieving tweets with minimal hassle. Unlike the traditional Twitter API, which comes with numerous restrictions and rate limits, this library leverages Twitter's own JavaScript-based frontend API. This ingenious approach allows for seamless extraction of tweet data without the need for API keys, tokens, or concerns about rate limits. It's fast, efficient, and accessible to everyone looking to gather tweet data.

Installation

Getting started with the Twitter Scraper is straightforward. You can install the library in your Go environment using the following command:

go get -u github.com/n0madic/twitter-scraper

Usage

Authentication

All methods within the Twitter Scraper now require user authentication. Here's how you can authenticate:

Login with Username and Password:
```
err := scraper.Login("username", "password")
```
For additional security measures such as two-factor authentication or email confirmation, you can add those details as well:
```
err := scraper.Login("username", "password", "email")
```
If a code for two-factor authentication is required:
```
err := scraper.Login("username", "password", "code")
```
You can check if you're logged in using:
```
scraper.IsLoggedIn()
```

Logout and Session Management:

To logout or clear the session:

scraper.Logout()

Sessions can be saved and restored using cookies to maintain authentication between restarts:

cookies := scraper.GetCookies()
// Save cookies as JSON
js, _ := json.Marshal(cookies)
f, _ = os.Create("cookies.json")
f.Write(js)

To load cookies:

f, _ := os.Open("cookies.json")
var cookies []*http.Cookie
json.NewDecoder(f).Decode(&cookies)
scraper.SetCookies(cookies)
scraper.IsLoggedIn()

Open Account Login:

If account credentials are an issue, simply login as a Twitter app:
```
err := scraper.LoginOpenAccount()
```

Retrieving Tweets

User's Tweets:

To get tweets from a specific user, use the following code:

scraper := twitterscraper.New()
err := scraper.LoginOpenAccount()
for tweet := range scraper.GetTweets(context.Background(), "Twitter", 50) {
    fmt.Println(tweet.Text)
}

This allows for fetching up to 50 tweets from the specified user.

Single Tweet:

To retrieve a single tweet using its ID:

tweet, err := scraper.GetTweet("1328684389388185600")
fmt.Println(tweet.Text)

Search Tweets by Query:

The library provides a method to search tweets using standard query operators. This feature requires authentication:
```
for tweet := range scraper.SearchTweets(context.Background(), "twitter scraper data -filter:retweets", 50) {
    fmt.Println(tweet.Text)
}
```
It stops searching after collecting 50 tweets.
Search Modes:

Multiple search modes are supported:
```
scraper.SetSearchMode(twitterscraper.SearchLatest)
```
Options include SearchTop, SearchLatest, SearchPhotos, SearchVideos, and SearchUsers.

Profile and Trends

Get Profile Information:

Retrieve profile data for a specific user:

profile, err := scraper.GetProfile("Twitter")
fmt.Printf("%+v\n", profile)

Search Profiles:

Search for user profiles by a query:

for profile := range scraper.SearchProfiles(context.Background(), "Twitter", 50) {
    fmt.Println(profile.Name)
}

Get Trending Topics:

To discover current trends:

trends, err := scraper.GetTrends()
fmt.Println(trends)

Advanced Features

Proxy Support:

The library supports HTTP and SOCKS5 proxies:

err := scraper.SetProxy("http://localhost:3128")

For SOCKS5:

err := scraper.SetProxy("socks5://localhost:1080")

Request Delays:

Introduce delays between API requests:
```
scraper.WithDelay(5)
```
Load Reply Timelines:

To include tweet replies in timelines:
```
scraper.WithReplies(true)
```

In summary, the Twitter Scraper brings unprecedented ease to retrieving Twitter data. It is a powerful alternative to the official API, eliminating typical obstacles and simplifying access to Twitter's corpus of information.