HttpResponseCache原理分析

Android

从Android4.0(API 14)开始，SDK源码中新增了一个类:android.net.http.HttpResponseCache.使用这个类可以很方便的对HTTP和HTTPS请求实现cache，所有的缓存逻辑再也不用自己写了，只要你使用HttpURLConnection或者HttpsURLConnection作为默认的网络请求库(也是Google官方建议使用的)，底层默认帮你实现的缓存的管理，不支持HttpClient。

根据developer文档介绍，开启HttpResponseCache很简单，只需要几行代码。

1
2
3
4
5
6
7
try {
   File httpCacheDir = new File(context.getCacheDir(), "http");
   long httpCacheSize = 10 * 1024 * 1024; // 10 MiB
   HttpResponseCache.install(httpCacheDir, httpCacheSize);
catch (IOException e) {
   Log.i(TAG, "HTTP response cache installation failed:" + e);
}

以上代码会在data/data/packagename/cache/下创建http文件夹，所有的http缓存文件默认都会存放到这个文件夹下，缓存文件夹最大为10M。
如果想兼容4.0以前的版本的话，可以用反射的方式调用。

1
2
3
4
5
6
7
8
9
try {
    File httpCacheDir = new File(context.getCacheDir(), "http");
    long httpCacheSize = 10 * 1024 * 1024; // 10 MiB
    Class.forName("android.net.http.HttpResponseCache")
            .getMethod("install", File.class, long.class)
            .invoke(null, httpCacheDir, httpCacheSize);
    catch (Exception httpResponseCacheNotAvailable) {
    }
}

开启完之后，如果服务端接口实现了http缓存协议，客户端发送http请求就会在http文件夹下产生缓存数据，下次再次请求的话，缓存如果还在有效期内，就会从缓存文件中读取。对缓存使用的控制可以通过设置不同的Cache-Control头部。

  • 如果不想使用本地缓存，直接请求服务器最新数据，比如下拉刷新，可以这么设置。

    1
    connection.addRequestProperty("Cache-Control", "no-cache");

  • 如果只想使用本地缓存，不去向服务器请求数据，可以这么设置。

    1
    connection.addRequestProperty("Cache-Control", "only-if-cached");

  • 如果既想使用本地缓存，又怕服务器数据有更新，需要服务器验证，可以这么设置。

    1
    connection.addRequestProperty("Cache-Control", "max-age=0");

  • 如果想设置缓存过期后的有效时间，可以这么设置。

    1
    2
    int maxStale = 60 * 60 * 24 * 28; // tolerate 4-weeks stale
    connection.addRequestProperty("Cache-Control", "max-stale=" + maxStale);

以前看Picasso的源码，发现如果使用HttpUrlConnection就会开启HttpResponseCache，Disk Cache交给HttpResponseCache处理。如果使用OkHttp作为网络请求库，就使用OkHttp自带的Disk Cache。今天感兴趣HttpResponseCache这块底层的实现逻辑，到源码里看一下。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
public static HttpResponseCache install(File directory, long maxSize) throws IOException {
    HttpResponseCache installed = getInstalled();
    if (installed != null) {
        // don't close and reopen if an equivalent cache is already installed
        DiskLruCache installedCache = installed.delegate.getCache();
        if (installedCache.getDirectory().equals(directory)
                && installedCache.maxSize() == maxSize
                && !installedCache.isClosed()) {
            return installed;
        } else {
            IoUtils.closeQuietly(installed);
        }
    }
    HttpResponseCache result = new HttpResponseCache(directory, maxSize);
    ResponseCache.setDefault(result);
    return result;
}
public static HttpResponseCache getInstalled() {
    ResponseCache installed = ResponseCache.getDefault();
    return installed instanceof HttpResponseCache ? (HttpResponseCache) installed : null;
}

初次设置，会根据传入的directory和maxSize生成一个HttpResponseCache，并设置成默认的java.net.ResponseCache，到这里就完成了。
看一下调用HttpURLConnection发送请求的时候是如何处理缓存的以及响应结果是如何处理的。

1
2
3
4
5
6
URL url = new URL("http://localhost:8080/xxx.json")
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setUseCaches(true);
if (connection.getResponseCode() == 200) {
}

这是使用HttpURLConnection发送请求最简单的用法，其实HttpURLConnection是一个继承于java.net.URLConnection的抽象类，

1
2
public abstract class HttpURLConnection extends URLConnection {
}

所有的方法都没有具体的实现，关键还是看一下调用URL.openConnection()生成的具体实例是谁。

1
2
3
public URLConnection openConnection() throws IOException {
    return streamHandler.openConnection(this);
}

看一下这个streamHandler是什么，找到初始化的地方。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
void setupStreamHandler() {
    // 省略部分代码
    // Fall back to a built-in stream handler if the user didn't supply one
    if (protocol.equals("file")) {
        streamHandler = new FileHandler();
    } else if (protocol.equals("ftp")) {
        streamHandler = new FtpHandler();
    } else if (protocol.equals("http")) {
        streamHandler = new HttpHandler();
    } else if (protocol.equals("https")) {
        streamHandler = new HttpsHandler();
    } else if (protocol.equals("jar")) {
        streamHandler = new JarHandler();
    }
    if (streamHandler != null) {
        streamHandlers.put(protocol, streamHandler);
    }
}

根据不同的协议类型，对应不同的Handler，这里就是HttpsHandler。所以，url.openConnection()最终调用的是HttpsHandler的openConnection()方法，到libcore下去看一下。

1
2
3
4
@Override
protected URLConnection openConnection(URL u) throws IOException {
    return new HttpURLConnectionImpl(u, getDefaultPort());
}

openConnection()方法最终返回了一个HttpURLConnectionImpl对象。所以抽象类HttpURLConnection的实现到这里就找到了，后面看看请求的过程，调用getResponseCode()的过程中究竟对缓存是如果操作的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
@Override 
public final int getResponseCode() throws IOException {
    return getResponse().getResponseCode();
}
private HttpEngine getResponse() throws IOException {
    initHttpEngine();
    if (httpEngine.hasResponse()) {
        return httpEngine;
    }
    while (true) {
        try {
            httpEngine.sendRequest();
            httpEngine.readResponse();
        } catch (IOException e) {
        }
        Retry retry = processResponseHeaders();
        if (retry == Retry.NONE) {
            httpEngine.automaticallyReleaseConnectionToPool();
            return httpEngine;
        }
        /*
         * The first request was insufficient. Prepare for another...
         */
        String retryMethod = method;
        OutputStream requestBody = httpEngine.getRequestBody();
        /*
         * Although RFC 2616 10.3.2 specifies that a HTTP_MOVED_PERM
         * redirect should keep the same method, Chrome, Firefox and the
         * RI all issue GETs when following any redirect.
         */
        int responseCode = getResponseCode();
        if (responseCode == HTTP_MULT_CHOICE || responseCode == HTTP_MOVED_PERM
                || responseCode == HTTP_MOVED_TEMP || responseCode == HTTP_SEE_OTHER) {
            retryMethod = HttpEngine.GET;
            requestBody = null;
        }
        if (requestBody != null && !(requestBody instanceof RetryableOutputStream)) {
            throw new HttpRetryException("Cannot retry streamed HTTP body",
                    httpEngine.getResponseCode());
        }
        if (retry == Retry.DIFFERENT_CONNECTION) {
            httpEngine.automaticallyReleaseConnectionToPool();
        }
        httpEngine.release(true);
        httpEngine = newHttpEngine(retryMethod, rawRequestHeaders,
                httpEngine.getConnection(), (RetryableOutputStream) requestBody);
    }
}

initHttpEngine()方法会生成全局的httpEngine对象，13行、14行看方法名分别是发送发送请求、读取响应，感觉逻辑应该在这里。先看一下sendRequest()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
public final void sendRequest() throws IOException {
    if (responseSource != null) {
        return;
    }
    prepareRawRequestHeaders();
    initResponseSource();
    if (responseCache instanceof HttpResponseCache) {
        ((HttpResponseCache) responseCache).trackResponse(responseSource);
    }
    /*
     * The raw response source may require the network, but the request
     * headers may forbid network use. In that case, dispose of the network
     * response and use a BAD_GATEWAY response instead.
     */
    if (requestHeaders.isOnlyIfCached() && responseSource.requiresConnection()) {
        if (responseSource == ResponseSource.CONDITIONAL_CACHE) {
            IoUtils.closeQuietly(cachedResponseBody);
        }
        this.responseSource = ResponseSource.CACHE;
        this.cacheResponse = BAD_GATEWAY_RESPONSE;
        RawHeaders rawResponseHeaders = RawHeaders.fromMultimap(cacheResponse.getHeaders());
        setResponse(new ResponseHeaders(uri, rawResponseHeaders), cacheResponse.getBody());
    }
    if (responseSource.requiresConnection()) {
        sendSocketRequest();
    } else if (connection != null) {
        HttpConnectionPool.INSTANCE.recycle(connection);
        connection = null;
    }
}

先看一下prepareRawRequestHeaders()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
private void prepareRawRequestHeaders() throws IOException {
    requestHeaders.getHeaders().setStatusLine(getRequestLine());
    if (requestHeaders.getUserAgent() == null) {
        requestHeaders.setUserAgent(getDefaultUserAgent());
    }
    if (requestHeaders.getHost() == null) {
        requestHeaders.setHost(getOriginAddress(policy.getURL()));
    }
    if (httpMinorVersion > 0 && requestHeaders.getConnection() == null) {
        requestHeaders.setConnection("Keep-Alive");
    }
    if (requestHeaders.getAcceptEncoding() == null) {
        transparentGzip = true;
        requestHeaders.setAcceptEncoding("gzip");
    }
    if (hasRequestBody() && requestHeaders.getContentType() == null) {
        requestHeaders.setContentType("application/x-www-form-urlencoded");
    }
    long ifModifiedSince = policy.getIfModifiedSince();
    if (ifModifiedSince != 0) {
        requestHeaders.setIfModifiedSince(new Date(ifModifiedSince));
    }
    CookieHandler cookieHandler = CookieHandler.getDefault();
    if (cookieHandler != null) {
        requestHeaders.addCookies(
                cookieHandler.get(uri, requestHeaders.getHeaders().toMultimap()));
    }
}

可以看到，从4.0开始，每一个请求底层都会默认开启gzip压缩和Keep-Alive连接复用。
再看一下initResponseSource()方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
private void initResponseSource() throws IOException {
    responseSource = ResponseSource.NETWORK;
    if (!policy.getUseCaches() || responseCache == null) {
        return;
    }
    CacheResponse candidate = responseCache.get(uri, method,
            requestHeaders.getHeaders().toMultimap());
    if (candidate == null) {
        return;
    }
    Map<String, List<String>> responseHeadersMap = candidate.getHeaders();
    cachedResponseBody = candidate.getBody();
    if (!acceptCacheResponseType(candidate)
            || responseHeadersMap == null
            || cachedResponseBody == null) {
        IoUtils.closeQuietly(cachedResponseBody);
        return;
    }
    RawHeaders rawResponseHeaders = RawHeaders.fromMultimap(responseHeadersMap);
    cachedResponseHeaders = new ResponseHeaders(uri, rawResponseHeaders);
    long now = System.currentTimeMillis();
    this.responseSource = cachedResponseHeaders.chooseResponseSource(now, requestHeaders);
    if (responseSource == ResponseSource.CACHE) {
        this.cacheResponse = candidate;
        setResponse(cachedResponseHeaders, cachedResponseBody);
    } else if (responseSource == ResponseSource.CONDITIONAL_CACHE) {
        this.cacheResponse = candidate;
    } else if (responseSource == ResponseSource.NETWORK) {
        IoUtils.closeQuietly(cachedResponseBody);
    } else {
        throw new AssertionError();
    }
}

第6行的responseCache是全局变量,

1
private final ResponseCache responseCache = ResponseCache.getDefault();

还记得我们开始缓存的install的方法吗？

1
2
HttpResponseCache result = new HttpResponseCache(directory, maxSize);
ResponseCache.setDefault(result);

先set，再get。所以这里的responseCache就是HttpResponseCache。initResponseSource()这里就是缓存的请求发送前缓存的处理逻辑。
第6行从本地缓存文件中读取缓存数据，没有缓存的话直接返回，否则对缓存数据进行一些列的处理和转化，第22行调用chooseResponseSource()判断缓存数据是否过期、可用与否。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
public ResponseSource chooseResponseSource(long nowMillis, RequestHeaders request) {
    /*
     * If this response shouldn't have been stored, it should never be used
     * as a response source. This check should be redundant as long as the
     * persistence store is well-behaved and the rules are constant.
     */
    if (!isCacheable(request)) {
        return ResponseSource.NETWORK;
    }
    if (request.isNoCache() || request.hasConditions()) {
        return ResponseSource.NETWORK;
    }
    long ageMillis = computeAge(nowMillis);
    long freshMillis = computeFreshnessLifetime();
    if (request.getMaxAgeSeconds() != -1) {
        freshMillis = Math.min(freshMillis,
                TimeUnit.SECONDS.toMillis(request.getMaxAgeSeconds()));
    }
    long minFreshMillis = 0;
    if (request.getMinFreshSeconds() != -1) {
        minFreshMillis = TimeUnit.SECONDS.toMillis(request.getMinFreshSeconds());
    }
    long maxStaleMillis = 0;
    if (!mustRevalidate && request.getMaxStaleSeconds() != -1) {
        maxStaleMillis = TimeUnit.SECONDS.toMillis(request.getMaxStaleSeconds());
    }
    if (!noCache && ageMillis + minFreshMillis < freshMillis + maxStaleMillis) {
        if (ageMillis + minFreshMillis >= freshMillis) {
            headers.add("Warning", "110 HttpURLConnection \"Response is stale\"");
        }
        if (ageMillis > TimeUnit.HOURS.toMillis(24) && isFreshnessLifetimeHeuristic()) {
            headers.add("Warning", "113 HttpURLConnection \"Heuristic expiration\"");
        }
        return ResponseSource.CACHE;
    }
    if (lastModified != null) {
        request.setIfModifiedSince(lastModified);
    } else if (servedDate != null) {
        request.setIfModifiedSince(servedDate);
    }
    if (etag != null) {
        request.setIfNoneMatch(etag);
    }
    return request.hasConditions()
            ? ResponseSource.CONDITIONAL_CACHE
            : ResponseSource.NETWORK;
}

逻辑不太复杂，如果设置了本次请求不需要缓存或者缓存不存在，直接返回ResponseSource.NETWORK；如果有缓存且没过期则返回ResponseSource.CACHE；如果缓存不再新鲜需要向服务器验证则返回ResponseSource.CONDITIONAL_CACHE，同时会更新Request的和缓存相关的头信息（37行——If-Modified-Since、39行——If-None-Match）。

initResponseSource()方法的23行会根据返回的ResponseSource进行逻辑处理，是直接返回缓存数据(setResponse())，还是带上缓存头部向服务器请求，还是不使用缓存向服务器请求新数据。

再回到sendRequest()方法中，第24行，根据上面对缓存的各种处理和判断，如果responseSource不等于ResponseSource.CACHE则发送网络请求，否则关闭Connect。

请求这块到这里就明白啦，再看一下响应的处理。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
public final void readResponse() throws IOException {
    if (hasResponse()) {
        return;
    }
    if (responseSource == null) {
        throw new IllegalStateException("readResponse() without sendRequest()");
    }
    if (!responseSource.requiresConnection()) {
        return;
    }
    if (sentRequestMillis == -1) {
        int contentLength = requestBodyOut instanceof RetryableOutputStream
                ? ((RetryableOutputStream) requestBodyOut).contentLength()
                : -1;
        writeRequestHeaders(contentLength);
    }
    if (requestBodyOut != null) {
        requestBodyOut.close();
        if (requestBodyOut instanceof RetryableOutputStream) {
            ((RetryableOutputStream) requestBodyOut).writeToSocket(requestOut);
        }
    }
    requestOut.flush();
    requestOut = socketOut;
    readResponseHeaders();
    responseHeaders.setLocalTimestamps(sentRequestMillis, System.currentTimeMillis());
    if (responseSource == ResponseSource.CONDITIONAL_CACHE) {
        if (cachedResponseHeaders.validate(responseHeaders)) {
            if (responseCache instanceof HttpResponseCache) {
                ((HttpResponseCache) responseCache).trackConditionalCacheHit();
            }
            // Discard the network response body. Combine the headers.
            release(true);
            setResponse(cachedResponseHeaders.combine(responseHeaders), cachedResponseBody);
            return;
        } else {
            IoUtils.closeQuietly(cachedResponseBody);
        }
    }
    if (hasResponseBody()) {
        maybeCache(); // reentrant. this calls into user code which may call back into this!
    }
    initContentStream(getTransferStream());
}

28行，如果服务器返回304 NOT_MODIFIED，说明服务端数据没更新、缓存可用，直接返回。看下41行的maybeCache()方法。

1
2
3
4
5
6
7
8
9
10
11
12
private void maybeCache() throws IOException {
    // Are we caching at all?
    if (!policy.getUseCaches() || responseCache == null) {
        return;
    }
    // Should we cache this response for this request?
    if (!responseHeaders.isCacheable(requestHeaders)) {
        return;
    }
    // Offer this request to the cache.
    cacheRequest = responseCache.put(uri, getHttpConnectionToCache());
}

第3行，如果设置了connection.setUseCaches(true)并且开启了HttpResponseCache才会保存缓存。第7行判断返回结果是否成功、服务器是否允许缓存，都通过的话最后把缓存数据保存到文件中去。到此关于缓存响应部分的处理就结束啦。

通过上面分析可以看到，Android底层关于缓存的处理严格遵循了HTTP的规范，通过设置不同的和缓存相关的头部信息进行缓存的判断和使用。这块和OKHttp底层的处理逻辑很像，感觉Android应该是借鉴了OkHttp代码的实现。