Skip to content

Develop #99

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
galaxyhm wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Develop #99

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a4b5389
쓸데없는 파일정리및 보기쉽게 바꾸는 작업중
galaxyhm May 8, 2023
c07211a
fixed untracked filed
galaxyhm May 8, 2023
104617d
deleted cached files
galaxyhm May 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# 프로젝트 이름 : ShortZip


## 프로젝트 기간
2023/04/14 ~ 2023/05/08 (24일)
## 개발 환경
Python 3.8.*

## 프로젝트 목적
뉴스기사 URL을 넣으면 요약을 해줘서 시간이 없는 분들이 빨리 읽을 수 있는 웹 사이트

## 사용 라이브러리 및 모델
[Kobart](https://github.com/SKT-AI/KoBART)
Expand All @@ -20,6 +22,7 @@ Python 3.8.*

## Example
<img src="https://i.imgur.com/yeVgryO.png" width="600px" height="400px" title="Site_Title"/>

<img src="https://i.imgur.com/JakqZvX.png" width="600px" height="400px" title="Site_Title"/>

not yet
Expand Down
3 changes: 2 additions & 1 deletion django_app/ARTICLE_SUMMARY2/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
db_connect.json
db_connect.json
fast_api.json
11 changes: 11 additions & 0 deletions django_app/ARTICLE_SUMMARY2/ARTICLE_SUMMARY2/settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,19 @@
# Database
# https://docs.djangoproject.com/en/3.2/ref/settings/#databases


FAST_API_HOST_ADDRESS = ''
FAST_API_HOST_PORT = 80

with open(BASE_DIR / 'fast_api.json') as fastapi:
fastapi_data_json = json.load(fastapi)
FAST_API_HOST_ADDRESS = fastapi_data_json['HOST']
FAST_API_HOST_PORT = fastapi_data_json['PORT']
fastapi.close()

with open(BASE_DIR / 'db_connect.json') as db_connect:
db_data = json.load(db_connect)
db_connect.close()
Comment on lines +88 to +92

Copilot AI May 20, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling .close() on the file handle inside a with open(...) block is redundant because with already handles closing.

Suggested change
fastapi.close()
with open(BASE_DIR / 'db_connect.json') as db_connect:
db_data = json.load(db_connect)
db_connect.close()
with open(BASE_DIR / 'db_connect.json') as db_connect:
db_data = json.load(db_connect)

Copilot uses AI. Check for mistakes.

DATABASES = {
'default': {
Expand Down
237 changes: 0 additions & 237 deletions django_app/ARTICLE_SUMMARY2/board/dowon_practice.ipynb
View file
Open in desktop

This file was deleted.

62 changes: 8 additions & 54 deletions django_app/ARTICLE_SUMMARY2/board/views.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@
import re
from . import models
from django.db.models import Q
import random
import pandas as pd
from django.conf import settings

# Create your views here.


Expand All @@ -38,9 +38,6 @@ def index(request):
# return render(request, 'board/join_membership.html', context)


# def summarize(request):
# return render()

@require_http_methods(['POST'])
def news_summarizae_request_ajax(request):
# print(request.body)
Expand All @@ -53,9 +50,9 @@ def news_summarizae_request_ajax(request):
url = regex.match(url)
print(url)

if not url :
if not url:
print('error 올바르지 않는 url')
# return 해서 어떻게 처리하셈
return JsonResponse({'status':'false', 'message': '잘못된 url'}, status=500)
url = url[0]

# DB에서 해당 url에 맞는 데이터 필드 가져옴.
Expand All @@ -70,7 +67,7 @@ def news_summarizae_request_ajax(request):
# 입력받은 URL을 가진 기사가 데이터베이스에 존재하는지 확인.
if models.NewsArticleInfo.objects.filter(url=url) :
# 해당 URL을 가진 기사가 데이터베이스에 존재하면, 수정일자를 비교하여 최신 버전인지 확인.
if models.NewsArticleInfo.objects.filter(Q(url=url) & Q(modify_date__isnull=True)) or models.NewsArticleInfo.objects.filter(Q(url=url) & Q(modify_date = crawl_data_dict.get('modify_date'))):
if (models.NewsArticleInfo.objects.filter(Q(url=url) & Q(modify_date__isnull=True)) and crawl_data_dict.get('modify_date') != -1 ) or models.NewsArticleInfo.objects.filter(Q(url=url) & Q(modify_date = crawl_data_dict.get('modify_date'))):

Copilot AI May 20, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] This conditional is hard to read. Consider extracting parts into well-named boolean variables or simplifying the logic for clarity.

Suggested change
if (models.NewsArticleInfo.objects.filter(Q(url=url) & Q(modify_date__isnull=True)) and crawl_data_dict.get('modify_date') != -1 ) or models.NewsArticleInfo.objects.filter(Q(url=url) & Q(modify_date = crawl_data_dict.get('modify_date'))):
is_new_article_with_modify_date = models.NewsArticleInfo.objects.filter(Q(url=url) & Q(modify_date__isnull=True)) and crawl_data_dict.get('modify_date') != -1
is_article_up_to_date = models.NewsArticleInfo.objects.filter(Q(url=url) & Q(modify_date=crawl_data_dict.get('modify_date')))
if is_new_article_with_modify_date or is_article_up_to_date:

Copilot uses AI. Check for mistakes.
news_article = models.NewsArticleInfo.objects.get(url=url)

return JsonResponse(
Expand All @@ -83,7 +80,7 @@ def news_summarizae_request_ajax(request):
}
)
# 최신 버전이 아닐 경우, 기사의 내용을 업데이트하고 요약을 다시 생성.
else :
else:
news_article = models.NewsArticleInfo.objects.get(url=url)
news_article.detail = crawl_data_dict['text']
news_article.modify_date = crawl_data_dict['modify_date']
Expand All @@ -96,7 +93,7 @@ def news_summarizae_request_ajax(request):

#정상적인 응답을 못 받았을때
if r.status_code != 200 :
pass
return JsonResponse({'status': 'false', 'message': 'fastapi가 응답 없음'}, status=500)
json_data = r.json()['message'][0].get('summary_text').strip()
news_article.summary = json_data
news_article.update()
Expand Down Expand Up @@ -168,16 +165,7 @@ def news_comments_request_ajax(request):
# comments가 0 인지 확인
if crawl_comment_list == [] :
return JsonResponse({'comments': []})
# crawl_comment_dict = dict()
# crawl_comment_dict['comments_data'] = crawl_comment_list

# DB에 저장및 조회 할 수 있는 코드 보류
# for comment in crawl_comment_list:
# comments_object.username = comment['userName']
# comments_object.contents = comment['contents']
# comments_object.sympathyCount = comment['sympathyCount']
# comments_object.antipathyCount = comment['antipathyCount']
# comments_object.save()



r = requests.post('http://13.208.62.74:8908/emotion/text/', data=json.dumps({'comments': crawl_comment_list}))
Expand All @@ -195,41 +183,7 @@ def news_comments_request_ajax(request):



@require_http_methods(['POST'])
def text_summarizae_request_ajax(request):


cond = {
"url" : "https://n.news.naver.com/article/018/0005472525"

}
jsonData = json.dumps(cond)
r = requests.post('http://localhost:10000/crawl/naver/', data=jsonData)
json_data = r.json()

print(json_data)
print(r.status_code)
# print(r.json())
# data = json.loads(m.body)
context = {
'result' : 'error '
}
# if data['url'] :
return JsonResponse(r.json())


def history_request_ajax(request):
return render(request, 'board/history.html')



def ajax_test(request) :

context = {'test' :'test'}

return JsonResponse(context)





23 changes: 1 addition & 22 deletions django_app/ARTICLE_SUMMARY2/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,4 @@
asgiref==3.6.0
beautifulsoup4==4.12.2
certifi==2022.12.7
charset-normalizer==3.1.0
Django==3.2.18
django-extensions==3.2.1
idna==3.4
lxml==4.9.2
mpmath==1.3.0
mysqlclient==2.1.1
networkx==3.1
numpy==1.24.3
packaging==23.1
pytz==2023.3
PyYAML==6.0
regex==2023.3.23
requests==2.29.0
six==1.16.0
soupsieve==2.4.1
sqlparse==0.4.4
sympy==1.11.1
tokenizers==0.13.3
typing_extensions==4.5.0
tzdata==2023.3
urllib3==1.26.15
lxml==4.9.2
48 changes: 25 additions & 23 deletions fastapi/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
from model import UrlItem, TextItem, CommentList
app = FastAPI()


#텍스트 전처리
def textProcessing( text: str) -> str:
text = text.replace(u'\n', u' ')
text = text.replace(u'\'', u' ')
Expand Down Expand Up @@ -59,29 +61,29 @@ def __init__(self):


# 입력 글자수 체크 밑 로직 미완성
def check_and_summarize(input_text, model_pipeline, max_length):
text_length = len(input_text)
text_start_pointer = 0
text_end_pointer = 0
return_text = []
while True:
if max_length < text_length - text_start_pointer:
if max_length > text_length - text_end_pointer:
text_end_pointer = text_length
else:
text_end_pointer = text_start_pointer + max_length
min_text = model_pipeline(input_text[text_start_pointer:text_end_pointer])
return_text.append(min_text)
print(min_text)
# return_text.append(model_pipeline.summarizer(input_text[text_start_pointer:text_start_pointer+1999]))
text_start_pointer = text_start_pointer+max_length-5
else:
# return_text.append(model_pipeline(input_text[text_start_pointer:]))
min_text = model_pipeline(input_text[text_start_pointer:])
return_text.append(min_text)
print(min_text)
# return_text.append(model_pipeline.summarizer(input_text[text_start_pointer:]))
return return_text
# def check_and_summarize(input_text, model_pipeline, max_length):
# text_length = len(input_text)
# text_start_pointer = 0
# text_end_pointer = 0
# return_text = []
# while True:
# if max_length < text_length - text_start_pointer:
# if max_length > text_length - text_end_pointer:
# text_end_pointer = text_length
# else:
# text_end_pointer = text_start_pointer + max_length
# min_text = model_pipeline(input_text[text_start_pointer:text_end_pointer])
# return_text.append(min_text)
# print(min_text)
# # return_text.append(model_pipeline.summarizer(input_text[text_start_pointer:text_start_pointer+1999]))
# text_start_pointer = text_start_pointer+max_length-5
# else:
# # return_text.append(model_pipeline(input_text[text_start_pointer:]))
# min_text = model_pipeline(input_text[text_start_pointer:])
# return_text.append(min_text)
# print(min_text)
# # return_text.append(model_pipeline.summarizer(input_text[text_start_pointer:]))
# return return_text


@app.post("/summarize/text")
Expand Down
13 changes: 3 additions & 10 deletions fastapi/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,4 @@
transformers>=4.28.1
fastapi>=0.95.1
pydantic>=1.10.7
pymysql>=1.0.3
mariadb>=1.1.6
requests>=2.28.2
numpy>=1.24.3
bs4>=0.0.1
beautifulsoup4>=4.12.2
lxml
pytorch
fastapi
uvicorn[standard]
torch
Loading