Spring Boot集成ip2region实现ip白名单

  • hblog

    2024-08-09 22:31:59
  • 数据库开发技术
  • 原创

1.什么是ip2region？

ip2region v2.0 - 是一个离线IP地址定位库和IP定位数据管理框架，10微秒级别的查询效率，提供了众多主流编程语言的 xdb 数据生成和查询客户端实现。

Ip2region 特性

1、标准化的数据格式

每个 ip 数据段的 region 信息都固定了格式：国家|区域|省份|城市|ISP，只有中国的数据绝大部分精确到了城市，其他国家部分数据只能定位到国家，后前的选项全部是0。

2、数据去重和压缩

xdb 格式生成程序会自动去重和压缩部分数据，默认的全部 IP 数据，生成的 ip2region.xdb 数据库是 11MiB，随着数据的详细度增加数据库的大小也慢慢增大。

3、极速查询响应

即使是完全基于 xdb 文件的查询，单次查询响应时间在十微秒级别，可通过如下两种方式开启内存加速查询：

  1. vIndex 索引缓存 ：使用固定的 512KiB 的内存空间缓存 vector index 数据，减少一次 IO 磁盘操作，保持平均查询效率稳定在10-20微秒之间。

  2. xdb 整个文件缓存：将整个 xdb 文件全部加载到内存，内存占用等同于 xdb 文件大小，无磁盘 IO 操作，保持微秒级别的查询效率。

4、IP 数据管理框架

v2.0 格式的 xdb 支持亿级别的 IP 数据段行数，region 信息也可以完全自定义，例如：你可以在 region 中追加特定业务需求的数据，例如：GPS信息/国际统一地域信息编码/邮编等。也就是你完全可以使用 ip2region 来管理你自己的 IP 定位数据。

2.代码工程

实验目标

根据来源ip 判断属于那个国家，然后做过滤

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0"         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">    <parent>        <artifactId>springboot-demoartifactId>        <groupId>com.etgroupId>        <version>1.0-SNAPSHOTversion>    parent>    <modelVersion>4.0.0modelVersion>
    <artifactId>ipfilterartifactId>
    <properties>        <maven.compiler.source>8maven.compiler.source>        <maven.compiler.target>8maven.compiler.target>    properties>    <dependencies>        <dependency>            <groupId>org.springframework.bootgroupId>            <artifactId>spring-boot-starter-webartifactId>        dependency>
        <dependency>            <groupId>org.springframework.bootgroupId>            <artifactId>spring-boot-autoconfigureartifactId>        dependency>        <dependency>            <groupId>org.springframework.bootgroupId>            <artifactId>spring-boot-starter-testartifactId>            <scope>testscope>        dependency>                <dependency>            <groupId>org.lionsoulgroupId>            <artifactId>ip2regionartifactId>            <version>2.6.4version>        dependency>        <dependency>            <groupId>org.projectlombokgroupId>            <artifactId>lombokartifactId>        dependency>        <dependency>            <groupId>com.alibaba.fastjson2groupId>            <artifactId>fastjson2artifactId>            <version>2.0.40version>        dependency>    dependencies>project>

controller

package com.et.ipfilter.controller;
import com.et.ipfilter.util.AddressUtils;import com.et.ipfilter.util.IPOfflineUtil;import com.et.ipfilter.util.IPOnlineUtil;import lombok.extern.slf4j.Slf4j;import org.springframework.util.StringUtils;import org.springframework.web.bind.annotation.RequestMapping;import org.springframework.web.bind.annotation.RestController;
import javax.servlet.http.HttpServletRequest;import javax.servlet.http.HttpServletResponse;import java.util.HashMap;import java.util.Map;
@RestController@Slf4jpublic class HelloWorldController {    @RequestMapping("/hello")    public Map<String, Object> showHelloWorld(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse,String ip){
        //fisrt get ip addr by offline file        //String ip = IPOfflineUtil.getIpAddr(httpServletRequest);        //analyze address        String addr = IPOfflineUtil.getAddr(ip);        if(StringUtils.isEmpty(addr)) {            //get addr by online service            ip = IPOfflineUtil.getIpAddr(httpServletRequest);            addr= AddressUtils.getRealAddressByIP(ip);            log.info("IP >> {},Address >> {}", ip, addr);         // you can filter by country or province        }        Map<String, Object> map = new HashMap<>();        map.put("msg", "HelloWorld");        map.put("ipaddr", addr);        return map;    }}

dto

package com.et.ipfilter.dto;
import lombok.AllArgsConstructor;import lombok.Data;import lombok.NoArgsConstructor;
    @Data    @NoArgsConstructor    @AllArgsConstructor    public  class CountryInfo {        private String query;        private String status;        private String country;        private String countryCode;        private String region;        private String regionName;        private String city;        private String zip;        private String lat;        private String lon;        private String timezone;        private String isp;        private String org;        private String as;
}

Util

离线查询ip来源，

下载离线IP定位库

离线数据库在项目的data文件夹下，名称为ip2region.db，其他2个文件是用于生成离线库的，可不用下载。

https://github.com/lionsoul2014/ip2region/tree/master/data/ip2region.xdb

下载到离线数据库后，我们需要读取这个数据库，我们可以放在项目的resources目录

package com.et.ipfilter.util;
import lombok.extern.slf4j.Slf4j;import org.lionsoul.ip2region.xdb.Searcher;import org.springframework.util.StringUtils;
import javax.servlet.http.HttpServletRequest;
/** * ip query */@Slf4jpublic class IPOfflineUtil {
    private static final String UNKNOWN = "unknown";
    protected IPOfflineUtil() {    }

    public static String getIpAddr(HttpServletRequest request) {        String ip = request.getHeader("X-Forwarded-For");
        if (!StringUtils.hasLength(ip) || UNKNOWN.equalsIgnoreCase(ip)) {            ip = request.getHeader("Proxy-Client-IP");        }        if (!StringUtils.hasLength(ip) || UNKNOWN.equalsIgnoreCase(ip)) {            ip = request.getHeader("WL-Proxy-Client-IP");        }        if (!StringUtils.hasLength(ip) || UNKNOWN.equalsIgnoreCase(ip)) {            ip = request.getHeader("HTTP_X_FORWARDED_FOR");        }        if (!StringUtils.hasLength(ip) || UNKNOWN.equalsIgnoreCase(ip)) {            ip = request.getHeader("HTTP_X_FORWARDED");        }        if (!StringUtils.hasLength(ip) || UNKNOWN.equalsIgnoreCase(ip)) {            ip = request.getHeader("HTTP_X_CLUSTER_CLIENT_IP");        }        if (!StringUtils.hasLength(ip) || UNKNOWN.equalsIgnoreCase(ip)) {            ip = request.getHeader("HTTP_CLIENT_IP");        }        if (!StringUtils.hasLength(ip) || UNKNOWN.equalsIgnoreCase(ip)) {            ip = request.getHeader("HTTP_FORWARDED_FOR");        }        if (!StringUtils.hasLength(ip) || UNKNOWN.equalsIgnoreCase(ip)) {            ip = request.getHeader("HTTP_FORWARDED");        }        if (!StringUtils.hasLength(ip) || UNKNOWN.equalsIgnoreCase(ip)) {            ip = request.getHeader("HTTP_VIA");        }        if (!StringUtils.hasLength(ip) || UNKNOWN.equalsIgnoreCase(ip)) {            ip = request.getHeader("REMOTE_ADDR");        }        if (!StringUtils.hasLength(ip) || UNKNOWN.equalsIgnoreCase(ip)) {            ip = request.getRemoteAddr();        }

        int index = ip.indexOf(",");        if (index != -1) {            ip = ip.substring(0, index);        }        return "0:0:0:0:0:0:0:1".equals(ip) ? "127.0.0.1" : ip;    }
    public static String getAddr(String ip) {        String dbPath = "D:\\IdeaProjects\\ETFramework\\ipfilter\\src\\main\\resources\\ip2region\\ip2region.xdb";        // 1、from dbPath load all xdb to memory。        byte[] cBuff;        try {            cBuff = Searcher.loadContentFromFile(dbPath);        } catch (Exception e) {            log.info("failed to load content from `%s`: %s\n", dbPath, e);            return null;        }
        // 2、usr cBuff create a query object base on memory。        Searcher searcher;        try {            searcher = Searcher.newWithBuffer(cBuff);        } catch (Exception e) {            log.info("failed to create content cached searcher: %s\n", e);            return null;        }        // 3、query        try {            String region = searcher.search(ip);            return region;        } catch (Exception e) {            log.info("failed to search(%s): %s\n", ip, e);        }        return null;    }}

在线查询访问ip

package com.et.ipfilter.util;
import com.et.ipfilter.dto.CountryInfo;import org.springframework.http.ResponseEntity;import org.springframework.web.client.RestTemplate;
import javax.annotation.PostConstruct;
/** * @author liuhaihua * @version 1.0 * @ClassName IPOnlineUtil * @Description todo * @date 2024/08/09/ 9:58 */
public class IPOnlineUtil {    private static RestTemplate restTemplate;
    private final RestTemplate template;
    public static final String IP_API_URL = "http://ip-api.com/json/";
    public IPOnlineUtil(RestTemplate restTemplate) {        this.template = restTemplate;    }
    /**     * init RestTemplate     */    @PostConstruct    public void init() {        setRestTemplate(this.template);    }
    /**     * init RestTemplate     */    private static void setRestTemplate(RestTemplate template) {        restTemplate = template;    }
    /**     * get country by ip     *     * @param ip     * @return     */    public static CountryInfo getCountryByIpOnline(String ip) {        ResponseEntity entity = restTemplate.getForEntity(                IP_API_URL + ip + "?lang=zh-CN",                CountryInfo.class        );        return entity.getBody();

    }}

IP 查询工具类

package com.et.ipfilter.util;
import org.lionsoul.ip2region.xdb.Searcher;import java.io.IOException; public class SearcherIPUtils {    public static String  getCachePosition(String ip) {        return SearcherIPUtils.getCachePosition("src/main/resources/ip2region/ip2region.xdb", ip, true);    }     public static String  getPosition(String dbPath,String ip,boolean format) {        // 1、create searcher object        Searcher searcher = null;        try {            searcher = Searcher.newWithFileOnly(dbPath);        } catch (IOException e) {            throw new RuntimeException(e);        }         // 2、query        try {            String region = searcher.search(ip);            if (format){                return region;            }            String[] split = region.split("\\|");            String s = split[0] + split[2] + split[3];            return s;        } catch (Exception e) {            throw new RuntimeException(e);        }
    }     /**     * @Description :     * @Author : mabo    */    public static String getIndexCachePosition(String dbPath, String ip, boolean format) {        Searcher searcher = null;        byte[] vIndex;        try {            vIndex = Searcher.loadVectorIndexFromFile(dbPath);        } catch (Exception e) {            throw new RuntimeException(e);        }        try {            searcher = Searcher.newWithVectorIndex(dbPath, vIndex);        } catch (Exception e) {            throw new RuntimeException(e);        }        try {            String region = searcher.search(ip);            if (format){                return region;            }            String[] split = region.split("\\|");            String s = split[0] + split[2] + split[3];            return s;        } catch (Exception e) {            throw new RuntimeException(e);        }    }     /**     * @Description :     * @Author : mabo     */    public static String  getCachePosition(String dbPath,String ip,boolean format) {        byte[] cBuff;        try {            cBuff = Searcher.loadContentFromFile(dbPath);        } catch (Exception e) {            throw new RuntimeException(e);        }         Searcher searcher;        try {            searcher = Searcher.newWithBuffer(cBuff);        } catch (Exception e) {            throw new RuntimeException(e);        }        try {            String region = searcher.search(ip);            if (format){                return region;            }            String[] split = region.split("\\|");            String s = split[0] + split[2] + split[3];            return s;        } catch (Exception e) {            throw new RuntimeException(e);        }     }}

package com.et.ipfilter.util;
import com.alibaba.fastjson2.JSONObject;import lombok.extern.slf4j.Slf4j;import org.springframework.util.StringUtils;
import java.io.BufferedReader;import java.io.IOException;import java.io.InputStreamReader;import java.net.ConnectException;import java.net.SocketTimeoutException;import java.net.URL;import java.net.URLConnection;
@Slf4jpublic class AddressUtils {
    public static final String IP_URL = "http://whois.pconline.com.cn/ipJson.jsp";     public static final String UNKNOWN = "XX XX";     public static String getRealAddressByIP(String ip) {        String address = UNKNOWN;         //if ip is inner ip,return        if (internalIp(ip)) {            return "inner IP";        }        if (true) {            try {                String rspStr = sendGet(IP_URL, "ip=" + ip + "&json=true" ,"GBK");                if (StringUtils.isEmpty(rspStr)) {                    log.error("get addr exception {}" , ip);                    return UNKNOWN;                }                JSONObject obj = JSONObject.parseObject(rspStr);                String region = obj.getString("pro");                String city = obj.getString("city");                return String.format("%s %s" , region, city);            } catch (Exception e) {                log.error("get addr exception {}" , ip);            }        }        return address;    }     public static String sendGet(String url, String param, String contentType) {        StringBuilder result = new StringBuilder();        BufferedReader in = null;        try {            String urlNameString = url + "?" + param;            log.info("sendGet - {}" , urlNameString);            URL realUrl = new URL(urlNameString);            URLConnection connection = realUrl.openConnection();            connection.setRequestProperty("accept" , "*/*");            connection.setRequestProperty("connection" , "Keep-Alive");            connection.setRequestProperty("user-agent" , "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;SV1)");            connection.connect();            in = new BufferedReader(new InputStreamReader(connection.getInputStream(), contentType));            String line;            while ((line = in.readLine()) != null) {                result.append(line);            }            log.info("recv - {}" , result);        } catch (ConnectException e) {            log.error("invoke HttpUtils.sendGet ConnectException, url=" + url + ",param=" + param, e);        } catch (SocketTimeoutException e) {            log.error("invoke HttpUtils.sendGet SocketTimeoutException, url=" + url + ",param=" + param, e);        } catch (IOException e) {            log.error("invoke HttpUtils.sendGet IOException, url=" + url + ",param=" + param, e);        } catch (Exception e) {            log.error("invoke HttpsUtil.sendGet Exception, url=" + url + ",param=" + param, e);        } finally {            try {                if (in != null) {                    in.close();                }            } catch (Exception ex) {                log.error("invoke in.close Exception, url=" + url + ",param=" + param, ex);            }        }        return result.toString();    }      /**     * check the ip is inner?     *     * @param ip     * @return     */    public static boolean internalIp(String ip) {        byte[] addr = textToNumericFormatV4(ip);        return internalIp(addr) || "127.0.0.1".equals(ip);    }     /**     * check the ip is inner?     *     * @param addr     * @return     */    private static boolean internalIp(byte[] addr) {        if (null==addr|| addr.length < 2) {            return true;        }        final byte b0 = addr[0];        final byte b1 = addr[1];        // 10.x.x.x/8        final byte SECTION_1 = 0x0A;        // 172.16.x.x/12        final byte SECTION_2 = (byte) 0xAC;        final byte SECTION_3 = (byte) 0x10;        final byte SECTION_4 = (byte) 0x1F;        // 192.168.x.x/16        final byte SECTION_5 = (byte) 0xC0;        final byte SECTION_6 = (byte) 0xA8;        switch (b0) {            case SECTION_1:                return true;            case SECTION_2:                return (b1 >= SECTION_3 && b1 <= SECTION_4);            case SECTION_5:                return (b1 == SECTION_6);            default:                return false;        }    }     /**     * change IPv4 to byte     *     * @param text     * @return byte     */    private static byte[] textToNumericFormatV4(String text) {        if (text.length() == 0) {            return new byte[0];        }         byte[] bytes = new byte[4];        String[] elements = text.split("\\.", -1);        try {            long l;            int i;            switch (elements.length) {                case 1:                    l = Long.parseLong(elements[0]);                    if ((l < 0L) || (l > 4294967295L)) {                        return new byte[0];                    }                    bytes[0] = (byte) (int) (l >> 24 & 0xFF);                    bytes[1] = (byte) (int) ((l & 0xFFFFFF) >> 16 & 0xFF);                    bytes[2] = (byte) (int) ((l & 0xFFFF) >> 8 & 0xFF);                    bytes[3] = (byte) (int) (l & 0xFF);                    break;                case 2:                    l = Integer.parseInt(elements[0]);                    if ((l < 0L) || (l > 255L)) {                        return new byte[0];                    }                    bytes[0] = (byte) (int) (l & 0xFF);                    l = Integer.parseInt(elements[1]);                    if ((l < 0L) || (l > 16777215L)) {                        return new byte[0];                    }                    bytes[1] = (byte) (int) (l >> 16 & 0xFF);                    bytes[2] = (byte) (int) ((l & 0xFFFF) >> 8 & 0xFF);                    bytes[3] = (byte) (int) (l & 0xFF);                    break;                case 3:                    for (i = 0; i < 2; ++i) {                        l = Integer.parseInt(elements[i]);                        if ((l < 0L) || (l > 255L)) {                            return new byte[0];                        }                        bytes[i] = (byte) (int) (l & 0xFF);                    }                    l = Integer.parseInt(elements[2]);                    if ((l < 0L) || (l > 65535L)) {                        return new byte[0];                    }                    bytes[2] = (byte) (int) (l >> 8 & 0xFF);                    bytes[3] = (byte) (int) (l & 0xFF);                    break;                case 4:                    for (i = 0; i < 4; ++i) {                        l = Integer.parseInt(elements[i]);                        if ((l < 0L) || (l > 255L)) {                            return new byte[0];                        }                        bytes[i] = (byte) (int) (l & 0xFF);                    }                    break;                default:                    return new byte[0];            }        } catch (NumberFormatException e) {            return new byte[0];        }        return bytes;    } }

以上只是一些关键代码，所有代码请参见下面代码仓库

代码仓库

  • https://github.com/Harries/springboot-demo(ipfilter）

3.测试

  • 启动 springBoot应用

  • 访问http://127.0.0.1:8088/hello?ip=52.220.113.16

  • 可以看到该ip属于新加坡，后续可以根据国家限制访问

4.引用

  • https://github.com/lionsoul2014/ip2region

请使用浏览器的分享功能分享到微信等