1/* Part of SWI-Prolog 2 3 Author: Jan Wielemaker 4 E-mail: J.Wielemaker@vu.nl 5 WWW: http://www.swi-prolog.org 6 Copyright (c) 2002-2016, University of Amsterdam, 7 VU University Amsterdam 8 All rights reserved. 9 10 Redistribution and use in source and binary forms, with or without 11 modification, are permitted provided that the following conditions 12 are met: 13 14 1. Redistributions of source code must retain the above copyright 15 notice, this list of conditions and the following disclaimer. 16 17 2. Redistributions in binary form must reproduce the above copyright 18 notice, this list of conditions and the following disclaimer in 19 the documentation and/or other materials provided with the 20 distribution. 21 22 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25 FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26 COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 28 BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 30 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 31 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 32 ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 33 POSSIBILITY OF SUCH DAMAGE. 34*/ 35 36:- module(http_client, 37 [ http_get/3, % +URL, -Reply, +Options 38 http_delete/3, % +URL, -Reply, +Options 39 http_post/4, % +URL, +In, -Reply, +Options 40 http_put/4, % +URL, +In, -Reply, +Options 41 http_patch/4, % +URL, +In, -Reply, +Options 42 http_read_data/3, % +Header, -Data, :Options 43 http_disconnect/1 % +What 44 ]). 45:- use_module(library(http/http_open)). 46:- use_module(library(uri)). 47:- use_module(http_header). 48:- use_module(http_stream). 49:- use_module(library(memfile)). 50:- use_module(library(lists)). 51:- use_module(library(error)). 52:- use_module(library(option)). 53 54:- meta_predicate 55 http_read_data(, , ). 56 57:- multifile 58 http_convert_data/4, % http_read_data plugin-hook 59 http:post_data_hook/3. 60 61:- predicate_options(http_get/3, 3, 62 [ pass_to(http_open/3, 3), 63 pass_to(http_read_data/3, 3) 64 ]). 65:- predicate_options(http_delete/3, 3, [pass_to(http_get/3, 3)]). 66:- predicate_options(http_post/4, 4, [pass_to(http_get/3, 3)]). 67:- predicate_options(http_put/4, 4, [pass_to(http_post/4, 4)]). 68:- predicate_options(http_read_data/3, 3, 69 [ to(any), 70 content_type(any), 71 form_data(oneof([form,mime])), 72 input_encoding(encoding), 73 on_filename(callable) 74 ]). 75 76 77/** <module> HTTP client library 78 79This library provides the four basic HTTP client actions: =GET=, 80=DELETE=, =POST= and =PUT=. In addition, it provides http_read_data/3, 81which is used by library(http/http_parameters) to decode =POST= data in 82server applications. 83 84This library is based on http_open/3, which opens a URL as a Prolog 85stream. The reply is processed by http_read_data/3. The following 86content-types are supported. Options passed to http_get/3 and friends 87are passed to http_read_data/3, which in turn passes them to the 88conversion predicates. Support for additional content types can be added 89by extending the multifile predicate http_client:http_convert_data/4. 90 91 - 'application/x-www-form-urlencoded' 92 Built in. Converts form-data into a list of `Name=Value` terms. 93 - 'application/x-prolog' 94 Built in. Reads a single Prolog term. 95 - 'multipart/form-data' 96 Processed if library(http/http_multipart_plugin) is loaded. This 97 format should be used to handle web forms that upload a file. 98 - 'text/html' | 'text/xml' 99 Processed if library(http/http_sgml_plugin) is loaded. See load_html/3 100 for details and load_xml/3 for details. The output is often processed 101 using xpath/3. 102 - 'application/json' | 'application/jsonrequest' 103 Processed if library(http/http_json) is loaded. The option 104 json_object(As) can be used to return a term json(Attributes) 105 (`As` is `term`) or a dict (`As` is `json`). 106*/ 107 108 /******************************* 109 * GET * 110 *******************************/ 111 112%! http_get(+URL, -Data, +Options) is det. 113% 114% Get data from a URL server and convert it to a suitable Prolog 115% representation based on the =|Content-Type|= header and plugins. 116% This predicate is the common implementation of the HTTP client 117% operations. The predicates http_delete/3, http_post/4 and 118% http_put/4 call this predicate with an appropriate 119% method(+Method) option and ---for http_post/4 and http_put/4--- 120% a post(+Data) option. 121% 122% Options are passed to http_open/3 and http_read_data/3. Other 123% options: 124% 125% - reply_header(-Fields) 126% Synonym for headers(Fields) from http_open/3. Provided for 127% backward compatibility. Note that http_version(Major-Minor) 128% is missing in the new version. 129 130http_get(URL, Data, Options) :- 131 headers_option(Options, Options1, Headers), 132 option(reply_header(Headers), Options, _), 133 http_open(URL, In, Options1), 134 delete(Headers, transfer_encoding(_), Headers1), 135 call_cleanup( 136 http_read_data(In, Headers1, Data, Options), 137 close(In)). 138 139headers_option(Options, Options1, Headers) :- 140 option(headers(Headers), Options), 141 !, 142 Options1 = Options. 143headers_option(Options, [headers(Headers)|Options], Headers). 144 145 146%! http_delete(+URL, -Data, +Options) is det. 147% 148% Execute a =DELETE= method on the server. Arguments are the same 149% as for http_get/3. Typically one should pass the option 150% status_code(-Code) to assess and evaluate the returned status 151% code. Without, codes other than 200 are interpreted as an error. 152% 153% @tbd Properly map the 201, 202 and 204 replies. 154% @see Implemented on top of http_get/3. 155 156http_delete(URL, Data, Options) :- 157 http_get(URL, Data, [method(delete)|Options]). 158 159 160%! http_post(+URL, +Data, -Reply, +Options) is det. 161% 162% Issue an HTTP =POST= request. Data is posted using 163% http_post_data/3. The HTTP server reply is returned in Reply, 164% using the same rules as for http_get/3. 165% 166% @see Implemented on top of http_get/3. 167 168http_post(URL, Data, Reply, Options) :- 169 http_get(URL, Reply, 170 [ post(Data) 171 | Options 172 ]). 173 174%! http_put(+URL, +Data, -Reply, +Options) 175% 176% Issue an HTTP =PUT= request. Arguments are the same as for 177% http_post/4. 178% 179% @see Implemented on top of http_post/4. 180 181http_put(URL, In, Out, Options) :- 182 http_post(URL, In, Out, [method(put)|Options]). 183 184%! http_patch(+URL, +Data, -Reply, +Options) 185% 186% Issue an HTTP =PATCH= request. Arguments are the same as for 187% http_post/4. 188% 189% @see Implemented on top of http_post/4. 190 191http_patch(URL, In, Out, Options) :- 192 http_post(URL, In, Out, [method(patch)|Options]). 193 194%! http_read_data(+Request, -Data, +Options) is det. 195% 196% Read data from an HTTP connection and convert it according to 197% the supplied to(Format) option or based on the =|Content-type|= 198% in the Request. The following options are supported: 199% 200% * to(Format) 201% Convert data into Format. Values are: 202% - stream(+WriteStream)) 203% Append the content of the message to Stream 204% - atom 205% Return the reply as an atom 206% - string 207% Return the reply as a string 208% - codes 209% Return the reply as a list of codes 210% * form_data(AsForm) 211% * input_encoding(+Encoding) 212% * on_filename(:CallBack) 213% These options are implemented by the plugin 214% library(http/http_multipart_plugin) and apply to processing 215% =|multipart/form-data|= content. 216% * content_type(+Type) 217% Overrule the content-type that is part of Request as a 218% work-around for wrongly configured servers. 219% 220% Without plugins, this predicate handles 221% 222% * 'application/x-www-form-urlencoded' 223% Converts form-data into a list of `Name=Value` terms. 224% * 'application/x-prolog' 225% Converts data into a Prolog term. 226% 227% @param Request is a parsed HTTP request as returned by 228% http_read_request/2 or available from the HTTP server's request 229% dispatcher. Request must contain a term input(In) that provides 230% the input stream from the HTTP server. 231 232http_read_data(Fields, Data, QOptions) :- 233 meta_options(is_meta, QOptions, Options), 234 memberchk(input(In), Fields), 235 ( http_read_data(In, Fields, Data, Options) 236 -> true 237 ; throw(error(failed(http_read_data), _)) 238 ). 239 240is_meta(on_filename). 241 242http_read_data(In, Fields, Data, Options) :- % Transfer-encoding: chunked 243 select(transfer_encoding(chunked), Fields, RestFields), 244 !, 245 setup_call_cleanup( 246 http_chunked_open(In, DataStream, []), 247 http_read_data(DataStream, RestFields, Data, Options), 248 close(DataStream)). 249http_read_data(In, Fields, Data, Options) :- 250 option(to(X), Options), 251 !, 252 ( X = stream(Stream) 253 -> ( memberchk(content_length(Bytes), Fields) 254 -> copy_stream_data(In, Stream, Bytes) 255 ; copy_stream_data(In, Stream) 256 ) 257 ; must_be(oneof([atom,string,codes]), X), 258 setup_call_cleanup( 259 new_memory_file(MemFile), 260 ( setup_call_cleanup( 261 open_memory_file(MemFile, write, Stream, 262 [encoding(octet)]), 263 ( memberchk(content_length(Bytes), Fields) 264 -> copy_stream_data(In, Stream, Bytes) 265 ; copy_stream_data(In, Stream) 266 ), 267 close(Stream)), 268 encoding(Fields, Encoding), 269 memory_file_to(X, MemFile, Encoding, Data0) 270 ), 271 free_memory_file(MemFile)), 272 Data = Data0 273 ). 274http_read_data(In, Fields, Data, _) :- 275 option(content_type(ContentType), Fields), 276 is_content_type(ContentType, 'application/x-www-form-urlencoded'), 277 !, 278 http_read_data(In, Fields, Codes, [to(string)]), 279 uri_query_components(Codes, Data). 280http_read_data(In, Fields, Data, Options) :- % call hook 281 ( select_option(content_type(Type), Options, Options1) 282 -> delete(Fields, content_type(_), Fields1), 283 http_convert_data(In, [content_type(Type)|Fields1], Data, Options1) 284 ; http_convert_data(In, Fields, Data, Options) 285 ), 286 !. 287http_read_data(In, Fields, Data, Options) :- 288 http_read_data(In, Fields, Data, [to(atom)|Options]). 289 290memory_file_to(atom, MemFile, Encoding, Data) :- 291 memory_file_to_atom(MemFile, Data, Encoding). 292memory_file_to(string, MemFile, Encoding, Data) :- 293 memory_file_to_string(MemFile, Data, Encoding). 294memory_file_to(codes, MemFile, Encoding, Data) :- 295 memory_file_to_codes(MemFile, Data, Encoding). 296 297 298encoding(Fields, utf8) :- 299 memberchk(content_type(Type), Fields), 300 ( sub_atom(Type, _, _, _, 'UTF-8') 301 -> true 302 ; sub_atom(Type, _, _, _, 'utf-8') 303 ), 304 !. 305encoding(_, octet). 306 307is_content_type(ContentType, Check) :- 308 sub_atom(ContentType, 0, Len, After, Check), 309 ( After == 0 310 -> true 311 ; sub_atom(ContentType, Len, 1, _, ';') 312 ). 313 314%! http_convert_data(+In, +Fields, -Data, +Options) is semidet. 315% 316% Multi-file hook to convert a HTTP payload according to the 317% _Content-Type_ header. The default implementation deals with 318% application/x-prolog. The HTTP framework provides 319% implementations for JSON (library(http/http_json)), HTML/XML 320% (library(http/http_sgml_plugin)) 321 322http_convert_data(In, Fields, Data, Options) :- 323 memberchk(content_type(Type), Fields), 324 is_content_type(Type, 'application/x-prolog'), 325 !, 326 ( memberchk(content_length(Bytes), Fields) 327 -> setup_call_cleanup( 328 ( stream_range_open(In, Range, [size(Bytes)]), 329 set_stream(Range, encoding(utf8)), 330 set_stream(Range, file_name('HTTP:DATA')) 331 ), 332 read_term(Range, Data, Options), 333 close(Range)) 334 ; set_stream(In, encoding(utf8)), 335 read_term(In, Data, Options) 336 ). 337 338%! http_disconnect(+Connections) is det. 339% 340% Close down some connections. Currently Connections must have the 341% value =all=, closing all connections. 342% 343% @deprecated New code should use http_close_keep_alive/1 from 344% library(http/http_open). 345 346http_disconnect(all) :- 347 http_close_keep_alive(_). 348 349%! http:post_data_hook(+Term, +Out, +Options) is semidet. 350% 351% Hook to extend the datatypes supported by the post(Data) option 352% of http_open/3. The default implementation supports 353% prolog(Term), sending a Prolog term as =|application/x-prolog|=. 354 355http:post_data_hook(prolog(Term), Out, HdrExtra) :- 356 setup_call_cleanup( 357 ( new_memory_file(MemFile), 358 open_memory_file(MemFile, write, Handle) 359 ), 360 ( format(Handle, 361 'Content-Type: application/x-prolog; charset=UTF-8~n~n', 362 []), 363 write_term(Handle, Term, 364 [ quoted(true), 365 ignore_ops(true), 366 fullstop(true), 367 nl(true) 368 ]) 369 ), 370 close(Handle)), 371 setup_call_cleanup( 372 open_memory_file(MemFile, read, RdHandle, 373 [ free_on_close(true) 374 ]), 375 http_post_data(cgi_stream(RdHandle), Out, HdrExtra), 376 close(RdHandle)).